RAG Eval

Regression Testing a RAG Pipeline

A RAG pipeline has a dozen knobs and each one silently affects the others. Without a suite that runs on every change, the second and third months of tuning undo the first month’s fixes, and nobody finds out until a user does.

The suite that works is boring: a small pinned corpus, retrieval-only assertions written as ranges, and a list of specific questions that must not break again.

Pin the corpus, or nothing is comparable

The first design decision is which corpus the tests run against, and the answer is not “production”.

Build a fixture corpus: a few hundred documents, checked into the repo or into object storage with a version tag, deliberately containing the structures your real corpus has — a table, a scanned page, two versions of one policy, a near-duplicate pair, a document with a long heading hierarchy. Small enough to index in seconds, weird enough to be representative.

A suite running against live production data fails for reasons unrelated to your change every time someone edits a document, and a test that fails for external reasons gets muted within a fortnight.

Pin everything else too: the embedding model and its version, the k, the distance metric, the prompt version, the judge version. Every unpinned parameter is a future mystery failure. Emit them all in the test output.

Three tiers of assertion

Tier 1 — invariants, no model calls. Facts about the pipeline’s plumbing that must hold regardless of quality:

  • every document in the fixture corpus produced at least one chunk
  • no chunk exceeds the embedding model’s input limit
  • chunk count and total token count are within a band of the recorded baseline
  • every chunk carries the metadata fields the prompt template expects
  • retrieval with a metadata filter returns only matching documents
  • the same query twice returns the same results against a frozen index
  • the assembled prompt for a fixed question is byte-identical to the recorded snapshot

That last one is the highest-value test in the suite and the one almost nobody has. Prompt assembly changes — a truncation rule, a separator, a metadata field silently dropping out — are invisible in every quality metric until they’re large, and a snapshot test catches them instantly.

Tier 2 — retrieval metrics against labels. Runs in seconds, no LLM calls, fully deterministic on a frozen index. This is the tier that goes in CI on every commit. Hit rate, recall@k, precision@k, per slice.

Tier 3 — generation and judge metrics. Slow, costly, non-deterministic. Nightly and pre-release, not per commit. The cadence argument is in which evals run on every commit.

Assert ranges, not values

assert recall_at_5 == 0.84 fails on the next index rebuild and teaches everyone to ignore the suite. Assert a floor, and a floor relative to the recorded baseline:

BASELINE = {"recall_at_5": 0.84, "precision_at_5": 0.61}
TOLERANCE = 0.03   # from measured run-to-run variance, not taste

def check(metrics):
    failures = []
    for name, base in BASELINE.items():
        got = metrics[name]
        if got < base - TOLERANCE:
            failures.append(f"{name}: {got:.3f} < {base:.3f} - {TOLERANCE}")
    return failures

Set TOLERANCE from measured variance — run the baseline twice and see how much it moves — rather than picking a number that feels safe. If your tolerance has to be wide to stop flapping, the real problem is noise, and the fix is a bigger set or a paired design: was that a real improvement or noise?

Also alert on suspicious improvements. A jump well outside tolerance usually means a bug — labels leaking into the index, the eval set overlapping the fixture corpus, a filter that stopped filtering. Treat a large unexplained gain as a failure until someone explains it.

Must-pass cases

Alongside the aggregate thresholds, keep a list of individual questions that must not regress. Each one comes from a real production failure that someone fixed.

{
  "id": "reg_007",
  "question": "can I return an item that arrived broken",
  "must_retrieve": ["policy-returns-v3#damaged"],
  "must_not_retrieve": ["policy-returns-v2#damaged"],
  "reason": "2026-06-14 — superseded policy version outranked current",
  "owner": "search"
}

must_not_retrieve earns its keep. Plenty of failures are not “the right chunk was missing” but “a wrong chunk was present” — a stale version, a near-duplicate, a document from the wrong tenant. Aggregate recall cannot express that; this field can.

These cases are the memory of the system. They accumulate, they never get deleted (only re-labelled when the corpus genuinely changes), and after a year they encode more institutional knowledge about your retrieval than any document does.

Same idea at the answer level: must_not_say entries in the reference set, as described in scoring answers against a reference.

Handling legitimate label changes

The awkward case: the suite fails and the suite is wrong. The corpus was re-chunked, document IDs changed, a policy was updated so the reference answer is stale.

Make relabelling an explicit, reviewable act rather than something that happens quietly when a test is annoying:

  • Labels live in version control, so a change is a diff someone approves.
  • Baseline updates go in their own commit, with the measured run that justifies them attached.
  • Re-chunks trigger a mapping step and a spot-check rather than a wholesale re-baseline — a re-chunk invalidates your eval set.
  • Nobody updates the baseline in the same commit as the change being measured. That commit is unreviewable by construction.

What CI should actually do

On every push: tier 1 invariants and tier 2 retrieval metrics against the fixture corpus. Target under two minutes. Fail on any invariant, on any aggregate metric below its floor, on any must-pass case, and on any declared slice below its own floor even when the aggregate is fine.

Nightly on the full eval set: tiers 1–3, results written to a durable artefact with the full configuration block, and a diff against yesterday posted where the team will see it.

Per release: the same plus a human review slice, and a look at the holdout you otherwise leave alone.

The artefact matters as much as the pass/fail. Six months of nightly results with configuration attached is how you answer “when did multi-source recall start dropping” — a question that is trivial with a history and unanswerable without one. Pair it with the production-side series in monitoring RAG quality in production, since some regressions never touch your code at all.

evaluationdebugging