Tagged “retrieval”
-
The Chunks Your Retriever Never Returns
Count how much of your index has ever been returned. The unreachable share is a defect list your eval set and your aggregate recall cannot see.
-
Your Labels Came From Your Old Retriever
A better retriever can score worse, because your labels only cover what the old one returned. Measure the unjudged rate and judge the delta.
-
Recall@k and What It Hides
Recall, precision, MRR and nDCG measure different things badly. What each one is blind to, and which to report.
-
Building a Retrieval Eval Set
Golden questions, labelled relevant documents, and the unanswerable cases. How to construct a set that predicts production behaviour.
-
A/B Testing a Retrieval Change
Online tests for RAG have no obvious success metric. Assignment units, behavioural proxies, guardrails, and what to do with 40 users.
-
Testing a New Embedding Model
Swapping embedders changes score scales, thresholds and truncation behaviour. The paired procedure, and the four ways the test goes wrong.
-
A Re-Chunk Invalidates Your Eval Set
Relevance labels point at chunk IDs that a re-chunk destroys. Label spans instead, and recover old labels by character overlap.
-
Evaluating Retrieval Before You Have Labels
No eval set yet? Four reference-free checks that find real retrieval failures in an afternoon — and the one thing none of them can measure.