RAG Eval

Building a Retrieval Eval Set

An eval set is a list of questions paired with the documents that should be retrieved to answer them. It’s the foundation everything else rests on, and most of them are built in a way that guarantees optimistic numbers.

Here’s how to build one that predicts something.

The shape

{
  "id": "q_014",
  "question": "how long do I have to return a damaged item",
  "relevant_doc_ids": ["policy-returns-v3#sec-2", "faq-shipping#damaged"],
  "answerable": true,
  "source": "support_logs_2026_06",
  "notes": "user phrasing; corpus says 'defective' not 'damaged'"
}

Five fields carry most of the value:

  • question — the query, in the phrasing a user would actually use.
  • relevant_doc_ids — every chunk or document that genuinely contains part of the answer. Multiple entries are normal and important.
  • answerable — false for questions the corpus cannot answer. You need these.
  • source — where the question came from. Lets you score subsets separately, which turns out to matter enormously.
  • notes — free text. Vocabulary mismatches recorded here become a fix list later.

Where the questions come from

Ranked by how much they predict production, best first:

1. Real user queries from logs. Unbeatable, if you have them. Sample across the whole distribution, not just the ones someone complained about — a set built from complaints measures your worst case only.

2. Questions from people who haven’t read the corpus. Pre-launch, this is the substitute. Give a domain expert a description of what the system covers and ask for the questions they’d actually ask. Do not let them browse the documents first; the moment they do, the questions inherit the corpus vocabulary and you’re back to the rigged case described in why your RAG demo works and production does not.

3. Support tickets, sales call notes, internal chat. Real questions in real phrasing, already written down.

4. LLM-generated from documents. The convenient option, and the weakest, for the same reason as writing them yourself: a model looking at a chunk produces a question phrased like that chunk. It’s not worthless — it’s a fast way to get coverage across a large corpus — but score it as its own subset and never let it be your only source. If you do use it, prompt for the question a user would ask rather than the question the passage answers, and accept that this only partly helps.

Aim for a mix. A hundred questions with sixty from real logs beats a thousand generated ones.

Labelling relevance

The labour is here, and it’s unavoidable.

For each question, find every chunk in the corpus containing part of the answer. Not the best one — all of them. Systems that surface a different-but-equally-valid source get unfairly penalised by single-label sets, and that penalty is invisible in the aggregate number.

Practical procedure: run your existing retriever at a generous k (say 50), have a human mark which of the returned chunks are relevant, and then — this is the part people skip — spot-check by searching the corpus directly with keywords, to catch relevant chunks your retriever never surfaced. Labelling only what your retriever returns builds a set that can never reveal what it’s missing.

Graded relevance — 2 for “fully answers,” 1 for “partially relevant,” 0 for “no” — costs a little more and unlocks nDCG, which is the right metric when some results are better than others rather than merely right or wrong.

Size and composition

Size: 100 questions is a usable floor for detecting large changes; 300–500 gives you enough resolution to trust small ones. Below 50, ordinary noise will exceed the effect of most changes you make.

Composition matters more than size. Deliberately include:

  • Unanswerable questions, 10–20%. Measures abstention. Without these you have no way to detect the confident-answer-to-an-impossible-question failure, which is your worst one.
  • Multi-document questions, where the answer requires combining two or more sources. These break naive top-k retrieval in ways single-source questions never reveal.
  • Vocabulary-mismatch questions, where the user’s word differs from the corpus’s word. Common in the wild, and the main thing dense retrieval is supposed to buy you — so measure whether it does.
  • Temporal questions, where two document versions exist and only the newer is correct. Retrieval scores both highly; only one is right.
  • Short and long queries. Two-word queries and paragraph-long ones behave differently.
  • The boring majority. Resist the urge to fill the set with hard cases. If 70% of production traffic is straightforward lookups, roughly that share of your eval set should be too, or your aggregate number describes a distribution you don’t have.

Keeping it honest over time

Freeze a holdout. Split the set: a development portion you look at constantly and a holdout you evaluate against rarely. Iterating against a single set until the number goes up is overfitting, and it’s easy to do accidentally over a few months of tuning.

Version it with the corpus. An eval set references document IDs. Reindex, re-chunk, or migrate the corpus and half those IDs may become invalid. Store the corpus version alongside the eval set, and treat a re-chunk as an event that requires relabelling — annoying, and less annoying than silently scoring against dead IDs.

Refresh from logs quarterly. Query distributions drift as users learn what the system can do, as the product changes, and as new documents land. A set built at launch describes launch.

Log the failures you fix. Every production failure worth investigating becomes a permanent eval case. Over a year this is how the set gets genuinely good — it accumulates exactly the cases your system got wrong, which is the definition of a useful regression suite.

Scoring it

Keep the evaluation loop trivial so it runs often:

def evaluate(retriever, eval_set, k=10):
    hits, recalls = 0, []
    for item in eval_set:
        if not item["answerable"]:
            continue                       # scored separately — see abstention
        got = {d.id for d in retriever.search(item["question"], k=k)}
        rel = set(item["relevant_doc_ids"])
        hits += bool(got & rel)
        recalls.append(len(got & rel) / len(rel))
    return {
        "hit_rate": hits / len(recalls),
        "recall_at_k": sum(recalls) / len(recalls),
    }

Two numbers, both retrieval-only, both computed without calling a language model — which means it runs in seconds and you can put it in CI.

Report them by subset, not just in aggregate. The overall figure hides everything interesting; recall on log-sourced questions versus generated ones, on multi-document versus single, on the temporal cases, is where the actionable information lives. What each of these metrics can’t see is covered in recall@k and what it hides.

And keep this set strictly for retrieval. Whether the model wrote a good answer from the context is a different measurement with a different set-up — see measuring faithfulness, not just answer quality.

evaluationretrieval