Evaluating Retrieval Before You Have Labels

Labelled relevance judgements are the foundation of retrieval evaluation and they take days of work. If you need a read on a system this week, there are reference-free checks that find genuine failures without any labels at all.

They share one hard limitation, stated up front so you don’t misuse them: without labels you can measure precision but not recall. You can see whether what came back is relevant. You cannot see what was missed.

1. Grade the retrieved chunks, per query

Take fifty real queries. For each, retrieve at your production k and have a judge rate each returned chunk as relevant or not to the query.

CHUNK_RELEVANT = """Query: {query}

Passage:
{chunk}

Could this passage contribute to answering the query?
Reply with exactly one word: YES or NO."""

Averaged over queries, the fraction of YES verdicts is a precision@k estimate. It’s the fastest useful number you can get, and it directly explains a very common complaint: answers that ramble or hedge, because seven of the ten chunks in the context were noise.

Two cautions. The judge is unvalidated at this point, so treat the number as directional and spot-check twenty verdicts by hand — the discipline is in validating an LLM judge before you trust it. And “could contribute” is a permissive standard that inflates the score; if you want a harsher read, ask whether the passage contains part of the answer.

2. Ask whether the context is sufficient

A better question than per-chunk relevance, because it captures the thing you care about: given only these chunks, could someone answer the query?

Three verdicts: sufficient, partially sufficient, insufficient. Run it over the same fifty queries.

This is the closest you can get to a recall proxy without labels, and the gap matters. A verdict of “insufficient” is strong evidence of a retrieval or corpus problem. A verdict of “sufficient” is weaker than it looks, because a judge reading three plausible chunks tends toward optimism, and because the material needed to answer well often exceeds the material needed to answer at all.

The distribution is the payoff. If a third of queries come back insufficient, you have a retrieval problem worth labelling properly. If almost none do and users still complain, your problem is downstream — generation, prompt assembly, or chunk quality — and you’ve just saved yourself a week of retrieval tuning.

3. Disagreement between two retrievers

Run each query through two different retrievers — your dense index and a plain keyword search over the same corpus is the easiest pair, since most stores support both.

Then look at the disagreement set: documents one retriever ranked highly and the other didn’t return at all.

This is the closest thing to a free recall check. Keyword search finds documents containing the query’s exact terms; dense retrieval finds paraphrases. Each surfaces things the other misses, and every item in the disagreement set that a human confirms as relevant is a document one of your retrievers failed to find. You now have evidence of misses without having labelled the corpus.

Read the disagreements rather than counting them. The patterns are diagnostic: keyword-only hits clustered on product codes, acronyms, and error strings mean your embeddings are losing exact identifiers. Dense-only hits on conversational phrasings mean keyword search is failing your users’ vocabulary. Both patterns tell you something specific about the fix.

4. Known-item probes

Pick twenty documents you know the corpus contains and care about. For each, write the query a user would use to find it — from the outside, without the document open, or you’ve written a query in the document’s own vocabulary and the test proves nothing.

Then check the rank at which that document appears.

This is a weak lower bound — you’re testing twenty hand-picked items, and it says nothing about the rest of the corpus. It’s also the fastest way to catch catastrophic problems: documents missing from the index entirely, an index that never got rebuilt after the last ingest, an embedding model mismatch between indexing and querying that quietly randomises everything, a filter silently excluding a whole document class.

Run known-item probes first. They take twenty minutes and they occasionally end the investigation immediately.

What none of this measures

Recall. Say it again, because reference-free numbers get quoted as if they were recall. Every technique above examines what came back. The document that was never retrieved and never appeared in either retriever’s list is invisible to all four.

Ranking quality within the relevant set. Position matters for RAG, and a precision estimate is position-blind — see recall@k and what it hides.

Multi-document sufficiency at scale. Check 2 partly covers this per query, but you can’t measure how systematically you fail multi-source questions without labels marking all the relevant sources.

Anything trendable. These are one-shot diagnostics. Reference-free scores drift with judge versions and query samples, so they make poor regression tests. The moment you want to answer “did this change help”, you need a fixed labelled set — building a retrieval eval set.

The order to do it in

An afternoon, in this sequence:

1. Known-item probes on twenty documents. Catches the catastrophic and the config-level.
2. Sufficiency judging on fifty real queries. Tells you whether retrieval is the bottleneck at all.
3. Per-chunk relevance on the same fifty. Tells you how much noise the generator is reading.
4. Retriever disagreement on the same fifty. Points at the specific class of misses.

Then use the output as the seed for a real eval set. The queries you ran are already collected; the insufficient ones and the confirmed disagreement hits are labelled examples you got for free; and the failure patterns tell you which categories the set needs to over-sample. Reference-free evaluation isn’t a substitute for labels — done in this order, it’s the cheapest way to know what to label.