Recall@k and What It Hides
Retrieval metrics are borrowed from information retrieval, where they’ve been well understood for decades. What’s new is the consumer: a language model with a finite context window and no ability to tell you which of the chunks you handed it were useless.
That changes which blind spots matter.
The four metrics
Hit rate@k — the fraction of questions where at least one relevant document appears in the top k. Binary per question.
Recall@k — of all the relevant documents for a question, what fraction appeared in the top k. Averaged over questions.
MRR (mean reciprocal rank) — 1 divided by the rank of the first relevant result, averaged. Rewards putting something relevant near the top.
nDCG@k — a graded, position-discounted score. Handles “this result is more relevant than that one” and penalises relevant results appearing lower. The most informative and the most work, since it needs graded labels.
What each one is blind to
Hit rate can’t see multi-document questions
Hit rate asks whether anything relevant showed up. For a question whose answer requires combining three documents, retrieving one of the three scores a full point — and the model then produces a confidently partial answer, which reads as complete because nothing signals the absence.
Hypothetical illustration: a set of 200 questions, 40 of which need two or more sources. Hit rate 0.91. Recall@10 on that 40-question subset, 0.52 — meaning about half the needed material never reached the model, on a fifth of the questions. Hit rate reported alone conceals this entirely. (Illustrative figures.)
Use hit rate only for single-source questions, and only as a first-pass sanity check.
Recall can’t see position or noise
Recall@10 treats a relevant document at rank 1 and at rank 10 identically. It also says nothing about the other nine results.
Both matter for RAG in a way they don’t for classic search, because the consumer is a model reading a concatenated prompt rather than a human scanning a list. Position affects how the model weights material. And nine irrelevant chunks alongside one relevant one is an actively harmful context — it consumes budget, and it gives the model plausible-looking material to synthesise from.
Report precision@k alongside recall. The ratio is the signal-to-noise of what you’re handing the model, and it’s the number that most often explains “retrieval looks fine but answers are bad.”
Recall@k hides the k
Recall@50 will always beat recall@10, and a stakeholder reading “recall 0.94” doesn’t know which you used. If your generator only consumes five chunks, recall@50 is measuring a system you don’t have.
Always write the k. Ideally report at the k you actually pass to the model, and if you use a reranker, report both stages: recall@50 for the retriever, precision@5 after reranking. Those are the two numbers that describe a two-stage pipeline honestly.
MRR can’t see the rest of the list
MRR only looks at the first relevant result. A system that puts one relevant document at rank 1 and buries four others at 40+ scores a perfect 1.0.
MRR is the right metric when exactly one document can answer the question and you’re feeding a single chunk to the model. For anything multi-source it’s actively misleading.
nDCG hides its own label quality
nDCG is the best of the four, and it’s completely dependent on graded relevance labels. If your grades were assigned quickly by one person, nDCG is a precise measurement of that person’s quick opinions.
Sample-check inter-annotator agreement if the number is load-bearing. If two labellers disagree on a third of the gradings, nDCG differences below a fairly wide margin mean nothing.
All of them are blind to chunk quality
The metric checks whether the right chunk ID was retrieved. It cannot check whether that chunk is comprehensible.
A chunk that begins mid-sentence, or that contains the relevant heading but not the paragraph beneath it, or that’s 400 tokens of navigation menu, will be marked correctly retrieved and will be useless to the model. Every retrieval metric will look healthy while the system produces nonsense.
This is the most common cause of “good retrieval scores, bad answers.” The check is not a metric — it’s reading fifty random chunks from your index by hand.
All of them are blind to unanswerable questions
Standard retrieval metrics are computed over questions that have relevant documents. Questions with none are typically excluded, since recall over an empty set is undefined.
So the failure mode where the system confidently answers something the corpus doesn’t cover is invisible to every metric in this post. It has to be measured separately, as abstention rate: of the questions you labelled unanswerable, what fraction produced a refusal rather than an answer.
Track it as a first-class number. For many systems it’s the one that matters most, and it’s the one nobody has.
What to actually report
Six numbers, and a note about what they exclude:
| Metric | At | Why |
|---|---|---|
| Recall@k | your generator’s k | Did the material reach the model |
| Precision@k | same k | How much noise came with it |
| Recall@k | retriever’s k, pre-rerank | Is the ceiling set by retrieval or by reranking |
| nDCG@k | your generator’s k | Position-aware quality, if you have graded labels |
| Abstention rate | — | Unanswerable subset only |
| Recall on multi-doc subset | your k | The subset every aggregate hides |
Plus a stated exclusion: “computed over 174 answerable questions; 26 unanswerable questions scored separately for abstention.”
Interpreting a change
Two cautions when a number moves.
Compute a confidence interval, or at minimum know your set size. On 100 questions, a recall change of a couple of points is within the noise you’d get by resampling the set. Bootstrapping over the eval set is a few lines of code and it will stop you shipping changes that did nothing.
Check whether the change moved the subsets differently. An aggregate that stays flat while the multi-document subset improves and the single-document subset degrades is a real finding hiding inside a null result — and it happens routinely with chunk-size changes.
Retrieval metrics only tell you what reached the model. What the model did with it is a separate measurement: measuring faithfulness, not just answer quality.