The Aggregate Score Hides the Bug

A single eval score is an average over a population of questions that behave nothing alike. It moves when the population’s composition changes, it stays flat when a whole segment breaks, and it cannot tell you where to work.

The fix is mechanical: define your slices once, report every metric per slice, and treat the aggregate as a headline rather than a finding.

How an aggregate goes wrong

Three distinct ways, worth distinguishing because they need different responses.

Dilution. A segment that’s 10% of your set gets much worse while everything else holds. The aggregate moves by a fraction of the damage and looks like noise. If your set has any minority segment you care about — one language, one document type, one tenant — the aggregate cannot protect it.

Reversal. The aggregate improves while several individual slices get worse. This happens when a change helps the largest slice a lot and hurts the small ones, and it happens more often than intuition suggests, because most retrieval changes are trades rather than gains. Hybrid retrieval that helps keyword-ish queries and hurts conversational ones is the canonical example.

Composition drift. Nobody changed the system; someone added forty easy questions to the eval set. The aggregate rises. Reported without a per-slice breakdown and a set version, this reads as a system improvement and is a bookkeeping artefact.

The slices that earn their place

Not every dimension is worth a column. These consistently are:

Question source. Log-sampled versus expert-written versus model-generated. The gap between them is the single most informative slice you can have, because it estimates how optimistic your set is. Generated questions score high; real user phrasings don’t. If you only ever report the mean of the mixture, you can improve your score by generating more questions.

Answerable versus unanswerable. Different metrics apply and averaging them is meaningless. Abstention behaviour is its own report — measuring whether your system knows when to refuse.

Single-source versus multi-source. Multi-document questions fail in ways single-source ones can’t reveal, and they’re usually a minority of the set, so dilution hides them perfectly.

Query length. Two-word queries and paragraph-long ones exercise retrieval differently. Three buckets is enough.

Document type. Answers grounded in tables, code, or scanned PDFs behave differently from prose. If your corpus is mixed, this slice frequently isolates an extraction problem masquerading as a retrieval problem.

Recency. Questions whose answer lives in a document that has multiple versions. Retrieval scores all versions highly and only the current one is correct.

Tenant, language, or product line. Whatever the axis is along which your users are not interchangeable. Small tenants are invisible in aggregates and vocal in support channels.

Doing it in the harness

Slicing should cost nothing per run, which means labelling questions with their slice keys once, at authoring time, rather than deriving them later.

def by_slice(results, eval_set, keys):
    """results: {question_id: score}. Returns {(key, value): (mean, n)}."""
    out = {}
    for key in keys:
        buckets = {}
        for item in eval_set:
            value = item.get(key)
            if value is None or item["id"] not in results:
                continue
            buckets.setdefault(value, []).append(results[item["id"]])
        for value, scores in buckets.items():
            out[(key, value)] = (sum(scores) / len(scores), len(scores))
    return out

Two properties of that function matter more than the arithmetic. It emits n alongside every mean, because a slice mean over six questions is not a number anyone should act on. And it iterates over declared keys, so the slice list is a fixed artefact in the repo rather than something an engineer chooses after seeing the results.

Pre-declare the slices

The reason to fix the list in advance: if you go looking for a favourable slice after the run, you’ll find one. With eight slice dimensions and a handful of values each you have dozens of comparisons, and some will look significant by chance alone. “Recall improved on long queries from the support-ticket source” is the kind of claim that survives no replication.

So: declare slices in the repo, report all of them every run, and require a slice to persist across two runs before it becomes a finding. If you spot an interesting new slice, add it permanently and re-run rather than reporting it once. The same discipline that keeps you from overfitting the whole set — when your eval set becomes training data — applies to slices.

Set a floor for slice size

Slices need a minimum n or they generate false alarms every week. A practical rule: don’t report a slice with fewer than about 30 questions, and don’t alert on one with fewer than about 50. Below that, ordinary sampling variation exceeds the effects you’re looking for — the reasoning is in was that a real improvement or noise?.

If a slice you care about is too small, the answer is to add questions to it, not to lower the bar. Deliberately over-sampling a minority segment in the eval set is fine and standard, as long as the aggregate is computed as a weighted average reflecting real traffic, or you skip the aggregate and only ever look at slices.

What to alert on

For a regression suite, the useful rule is: fail on aggregate regressions, and fail on slice regressions even when the aggregate improves.

That second clause is the point of the whole exercise. A change that lifts the mean while dropping recall on multi-source questions by a visible margin should stop the build and require an explicit decision, not sail through because the headline is green. Wire it in as part of regression testing a RAG pipeline.

Keep the per-slice history, too. Trends per slice are far more legible than a single line — a segment that has drifted down for six weeks is obvious in its own series and invisible in the mean.

Reporting it upward

Stakeholders will ask for one number. Give them one number and one table, and put the table first in the document even if the number goes in the summary line.

The sentence that makes the table land: “the average is 0.86, and it’s 0.93 on the questions we wrote and 0.71 on the ones users actually asked.” (Illustrative figures.) That framing is honest, it’s usually true, and it converts a vague quality conversation into a decision about which slice to fix.