When Your Eval Set Becomes Training Data

You never trained on your eval set, and you overfit it anyway. Every configuration choice made because it raised the number is a gradient step, taken by hand, and after a few hundred of them the score describes your set rather than your system.

The symptom is specific and recognisable: the eval score has improved steadily for a quarter and user complaints haven’t changed.

The five leakage paths

Manual hill-climbing. The main one. Chunk size, k, overlap, prompt wording, threshold, reranker depth — each chosen by trying options and keeping what scored best on the same 200 questions. Individually reasonable; collectively, fitting to the sample. Nothing about this requires bad practice, only repetition.

Prompt tuning against the set. Prompts get iterated hardest, and prompt space is enormous, so this is where the fitting concentrates. A prompt refined over fifty runs on one set encodes the quirks of those questions: their phrasing, their length, their politeness.

Questions generated from the corpus. A question a model wrote while looking at a chunk is phrased in that chunk’s vocabulary, so retrieval finds it easily. A set built this way is not merely optimistic — it is unrepresentative in the direction your system is best at, and improvements measured on it need not transfer at all.

Fixing the specific case instead of the class. A question fails, someone adds a synonym mapping or a metadata boost that makes that question pass. The score rises by one question and nothing generalises. Legitimate when it’s a genuine class of query; a smell when the fix mentions the eval question’s exact terms.

Judge-shaped answers. If the metric is judge-derived, you are also fitting the judge’s biases. Tune long enough and your answers become longer, more heavily hedged, more citation-dense — the things judges reward — without becoming more useful. This is Goodhart’s law with a specific mechanism, and it’s why the human review slice can’t be dropped: writing a rubric two reviewers agree on.

The holdout, and the discipline it needs

Split the set at construction: a development portion you use freely, and a holdout you evaluate against rarely.

The split has to be by question, stratified so both halves carry the same mix of sources, difficulty and slice values. A holdout that is accidentally easier than the dev set tells you nothing useful in either direction.

The discipline is the hard part, and three rules cover it:

• Look at the holdout on a schedule, not when a change looks promising. Per release, or monthly. Frequency is what converts a holdout into a second dev set.
• Never tune on it. Not even once, not even “just to check which of these two”. The moment a decision is made on holdout evidence, it’s a dev set with extra steps.
• Count your looks. Record every holdout evaluation with a date and reason. Twenty looks in a quarter means you have been tuning against it slowly, whatever anyone intended.

Nothing here needs tooling beyond a text file, and the text file is what makes the rule enforceable in six months when everyone has forgotten.

Read the gap, not just the score

The number that detects overfitting is the difference between dev and holdout performance, tracked over time.

A gap that is stable is fine — it’s an artefact of the split. A gap that widens across months is the signature: dev keeps improving while holdout flattens, which is precisely what fitting to the dev sample produces.

Watch two other gaps for the same reason:

• Generated versus log-sourced questions. If the log-sourced slice stops improving while the generated slice climbs, you’re improving on questions nobody asks. Slice reporting makes this visible for free — the aggregate score hides the bug.
• Offline versus online. The end of the chain. If offline improves and online metrics don’t, the offline set has stopped predicting: why offline gains vanish online.

Keep the set alive

A frozen set becomes unrepresentative even without any tuning, because users, corpus and product all move.

Add from logs on a schedule. Quarterly is a reasonable default. Sample across the whole query distribution, not just complaints; a set built from complaints measures your worst case only.

Add every investigated production failure. This is how the set gets genuinely good, and it needs one guard: add the class of failure, not just the exact query, or you accumulate a set of hyper-specific cases you’ve already fixed.

Retire dead questions. Questions whose documents were deleted, or whose answer changed, either get relabelled or become abstention cases.

Rotate a portion of the dev set into the archive as you add. Questions you’ve stared at fifty times have limited remaining information.

Version everything. Set version, corpus version, chunking and model versions, judge prompt version. A score without them is not comparable to any other score, and comparing incomparable numbers is how composition drift gets read as progress — see regression testing a RAG pipeline.

Contamination, the version that isn’t your fault

Two subtler cases worth checking once:

Eval questions inside the corpus. If your corpus includes support transcripts, FAQ pages, or internal wiki entries, and your eval questions were drawn from those same sources, retrieval may be finding the question-and-answer pair verbatim. That’s a trivially easy retrieval task and it doesn’t resemble production. Search the corpus for your eval questions’ distinctive phrasings; if a question exists in the corpus, either remove it or label it as its own slice.

Prompt examples drawn from the eval set. Few-shot examples taken from eval questions make those questions unrepresentatively easy. Keep example pools and eval sets strictly disjoint, and check it, because this drifts in as prompts get edited.

The honest report

Report dev and holdout side by side, with the number of holdout looks and the count of configurations tried since the last look.

“Dev faithfulness 0.91, holdout 0.84, third holdout evaluation this quarter, roughly forty configurations tried since the last one.” (Illustrative figures.) That last clause is the one nobody volunteers and the one that tells a reader how much of the dev gain to believe.