Tagged “evaluation”
-
Measuring a Prompt Change With Retrieval Held Still
Prompt edits only touch generation, so let retrieval vary and you measure noise. Freeze the contexts, compare paired, watch the judge's style bias.
-
Reproducing a Bad Answer Somebody Sent You
A screenshot is not a bug report. Three replay modes that turn one complaint into a reproducible case, and what a failure to reproduce tells you.
-
Testing That Retrieval Respects Permissions
Every quality metric can look perfect while retrieval leaks documents across identities. Test it as an invariant with identities in the eval set.
-
Your Labels Came From Your Old Retriever
A better retriever can score worse, because your labels only cover what the old one returned. Measure the unjudged rate and judge the delta.
-
Thumbs Down Is a Weak Label
User feedback is biased, sparse, and silent on confident wrong answers. Use it as a sampler, not a metric — and instrument it better.
-
Monitoring RAG Quality in Production
Live traffic has no labels. Eight signals you can watch anyway, plus scheduled canaries that catch config breaks within minutes.
-
The Five Places a RAG Pipeline Breaks
A bad answer implicates one of five stages. A bisection procedure that finds the responsible one in about ten minutes.
-
Measuring Faithfulness, Not Just Answer Quality
An answer can be good and ungrounded at the same time. How to check every claim against the retrieved context, and where LLM judges fail.
-
Building a Retrieval Eval Set
Golden questions, labelled relevant documents, and the unanswerable cases. How to construct a set that predicts production behaviour.
-
Why Your RAG Demo Works and Production Does Not
The demo and production differ in five specific ways, all of which favour the demo. Each one is measurable before you ship.
-
A Failure Taxonomy for RAG Answers
Read fifty bad answers, code each one, count the codes. The starter codebook and the procedure that turns anecdotes into a work queue.
-
A/B Testing a Retrieval Change
Online tests for RAG have no obvious success metric. Assignment units, behavioural proxies, guardrails, and what to do with 40 users.
-
Why Offline Gains Vanish Online
Seven reasons an eval-set improvement doesn't reach users, and how to shrink the gap by evaluating through the production path.
-
Detecting Regressions When the Model Updates
A hosted model changes under you and your quality moves with no deploy. Version pinning, scheduled canaries, and the judge that drifts too.
-
When Your Eval Set Becomes Training Data
Tune against one set for three months and the score stops predicting production. Five leakage paths, and the holdout discipline that catches them.
-
Testing a New Embedding Model
Swapping embedders changes score scales, thresholds and truncation behaviour. The paired procedure, and the four ways the test goes wrong.
-
A Re-Chunk Invalidates Your Eval Set
Relevance labels point at chunk IDs that a re-chunk destroys. Label spans instead, and recover old labels by character overlap.
-
Which Evals Run on Every Commit
Judge-based evaluation is too slow and costly for CI. A four-tier cadence that keeps fast feedback fast and expensive checks meaningful.
-
Regression Testing a RAG Pipeline
A pipeline with no regression suite gets worse in ways nobody notices. Fixture corpora, range assertions, and must-pass cases in CI.
-
Was That a Real Improvement or Noise?
Four sources of variance sit between your change and your eval score. Paired comparison and bootstrapping tell you which deltas to believe.
-
The Aggregate Score Hides the Bug
One number over a mixed eval set averages your best and worst behaviour into a figure that describes neither. Which slices to report, always.
-
Measuring Whether Your System Knows When to Refuse
Refusal is a classification problem with two error types. How to build the unanswerable set, score both cells, and catch fake refusals.
-
Evaluating Retrieval Before You Have Labels
No eval set yet? Four reference-free checks that find real retrieval failures in an afternoon — and the one thing none of them can measure.
-
Validating an LLM Judge Before You Trust It
A judge is an instrument with unknown error until you measure it. The labelled probe set, the bias checks, and the re-validation triggers.
-
Writing a Rubric Two Reviewers Agree On
Human review is only useful if two people reading the same answer give the same score. Rubric design, anchors, and measuring agreement.
-
Scoring Answers Against a Reference
Exact match and embedding similarity both fail on long-form answers. Key-point coverage is the reference metric that survives paraphrase.
-
Answer Relevance Is Not Answer Correctness
An answer can be true, grounded, and still not answer the question. How to measure relevance separately, and why holistic scores miss it.
-
Component or End-to-End: Which Evaluation Answers Which Question
End-to-end scores tell you whether users are served. Component scores tell you where to fix it. Neither substitutes for the other.