RAG Eval

Your Labels Came From Your Old Retriever

You swap dense-only retrieval for a hybrid setup, run the eval set, and recall@10 goes down: 0.81 to 0.74 (hypothetical figures). You read twenty of the questions that got worse and the new system’s results look better to you than the old system’s. Both observations are correct.

The metric is measuring your label coverage, not your retrieval.

Where the labels came from

Almost every retrieval eval set in existence was labelled by the procedure in building a retrieval eval set: run the retriever you have at a generous k, show a human what came back, mark what’s relevant. That procedure is the only affordable one — nobody reads a 400,000-chunk corpus per question — and it has a property that matters enormously the first time you compare two systems.

The set of documents that have a label is a sample of the corpus chosen by the system under test.

Everything outside that sample is unjudged. Your scoring code cannot tell unjudged from judged-irrelevant, because both are simply absent from relevant_doc_ids, so it counts them as misses. The old retriever’s good results are all inside the pool. A challenger’s genuinely-good-but-novel results are outside it, and get scored as noise.

The bias is asymmetric, and it always points the same way: toward whichever system built the pool. This is the mechanism behind a large share of “our fancy new retriever didn’t help” results, and it is invisible in the aggregate number.

Measure the unjudged rate first

Before you believe any cross-system delta, compute, for each system, the fraction of its top-k results that carry no label at all:

def unjudged_rate(results, labels, k):
    seen = judged = 0
    for qid, docs in results.items():
        known = labels[qid]["judged_ids"]      # relevant AND labelled-irrelevant
        for d in docs[:k]:
            seen += 1
            judged += d.id in known
    return 1 - judged / seen

Note judged_ids, not relevant_doc_ids. If your labelling only recorded the positives, you cannot compute this at all — which is itself worth fixing, because “this chunk was looked at and rejected” is information you paid a human for and threw away.

Read the two rates together:

Old system Challenger Reading
low low Comparison is meaningful.
low high Challenger is retrieving outside the pool. Delta is uninterpretable.
high high Pool is too small or the corpus changed. Relabel before comparing anything.

A challenger whose unjudged rate is several times the incumbent’s has not been evaluated. It has been penalised for being different.

Judge the delta, not the corpus

The fix is cheap because the work is bounded by the difference between the two systems, not by either system’s output.

  1. Run both systems at your reporting k (or wider — pool at 20 and report at 5).
  2. Take the union of returned document IDs per question.
  3. Subtract everything already judged.
  4. Send only that remainder to a human, question by question, with the question text and the chunk text side by side.
  5. Merge the new verdicts — positive and negative — into the label file, and rescore both systems.

In practice the remainder is a small fraction of the union, because two retrievers over the same corpus agree on a lot. That is the whole trick: the incremental judging cost of adding a system to the pool falls as the pool grows.

Record provenance while you’re there:

{
  "id": "q_014",
  "relevant_doc_ids": ["policy-returns-v3#sec-2"],
  "judged_ids": ["policy-returns-v3#sec-2", "faq-shipping#damaged", "policy-returns-v2#sec-2"],
  "pool": [
    {"system": "dense-v1", "k": 50, "date": "2026-06-02"},
    {"system": "hybrid-v2", "k": 50, "date": "2026-07-29"}
  ]
}

The pool block is the field that stops this happening twice. Six months later it tells you which systems the set can fairly compare and which one it was born from.

The cheap approximation, and its own bias

If you cannot get judging time, score on judged results only: drop unjudged documents from each result list, then compute recall and precision over what remains. This is the standard trick for reusing an old pool, and it removes the penalty for novelty.

It also introduces its own distortion. A system whose top results are mostly unjudged gets scored on a short, filtered list that may not resemble what the user would see, and the metric quietly rewards agreeing with the pool. Use it as a sanity check that runs alongside the raw number, never as the headline, and always print the unjudged rate next to it.

Near-duplicates make this worse

If your corpus contains the same content in several places — a policy in the handbook and in the FAQ, a page and its PDF export, three tenants with the same template — then a challenger that returns the other copy is marked wrong for retrieving an identical passage.

Handle it in the labels rather than in the retriever: give each label an equivalence class, and score a hit if any member of the class was retrieved.

"relevant_classes": [
  ["policy-returns-v3#sec-2", "faq-shipping#damaged", "handbook-2026#p41"]
]

Keep the version-supersession cases out of the class. Two copies of the current policy are equivalent; the current policy and last year’s are not, and collapsing them destroys exactly the temporal case your set exists to catch.

What the numbers license you to say

A pooled eval set is a good instrument for one job and a poor one for another.

It can measure regressions in the system that built it. Nothing about pooling breaks that: you’re asking whether today’s version still finds what yesterday’s version found, and the pool covers yesterday’s version by construction. This is why the CI tier in which evals run on every commit works fine on an old set.

It cannot fairly measure a challenger until the challenger is in the pool. So: never quote a cross-system recall delta without the two unjudged rates beside it, and treat “the new retriever scored lower” as an unresolved question rather than a result until you’ve read the delta by hand.

And when you do get a clean paired comparison, it still has to clear the noise floor before it means anything — see was that a real improvement or noise?. The same discipline applies to an embedding swap, where every document’s vector changes at once: testing a new embedding model.

evaluationretrieval