RAG Eval

Was That a Real Improvement or Noise?

You changed the retriever, recall@5 went from 0.81 to 0.84, and the question is whether you learned anything. On a 150-question eval set with a non-deterministic generator and an LLM judge in the loop, a three-point move is well within the range that repeating the same run can produce.

Two techniques settle it: compare the arms on the same questions, and bootstrap the per-question scores to get an interval. Both are a few lines of code and they change which results you act on.

The four sources of variance

Sampling variance of the eval set. Your set is a sample of the question space, and its mean is an estimate. For a proportion-like metric (hit rate, faithfulness pass rate), the standard error is roughly sqrt(p * (1 - p) / n). On 100 questions around p = 0.8 that’s about 0.04, so a naive 95% interval spans roughly eight points either side. Most reported “improvements” are smaller than this.

Generation non-determinism. Same prompt, same context, different answer. Temperature zero reduces it and does not eliminate it, because hosted inference varies for reasons outside your control.

Judge non-determinism. The same claim gets a different verdict on a re-run. Measure the flip rate directly and treat it as a floor on detectable change — the procedure is in validating an LLM judge before you trust it.

Index non-determinism. Approximate nearest-neighbour search is approximate. Rebuild the index, or query a differently-sharded replica, and borderline results at the tail of the top-k move. Ordinarily small; occasionally the whole effect you thought you measured.

These stack. Retrieval-only metrics on a frozen index carry only the first, which is why the cheap tier of a regression suite is also the most trustworthy one.

Pair the comparison

The single highest-value change to how you compare two systems: run both arms over the same questions and compare per question, not in aggregate.

Unpaired comparison asks whether one mean differs from another, and the variance of that difference includes all the between-question variance — some questions are simply harder, and that spread dwarfs your effect. Paired comparison cancels it, because each question is its own control.

What you report changes shape, and improves:

def paired(before, after):
    """before/after: {question_id: score}. Returns wins, losses, ties."""
    ids = set(before) & set(after)
    wins = sum(after[i] > before[i] for i in ids)
    losses = sum(after[i] < before[i] for i in ids)
    return wins, losses, len(ids) - wins - losses

“Recall improved on 22 questions, worsened on 9, unchanged on 119” is a far more informative sentence than “+0.03”. It tells you the effect is real and it tells you the change is a trade — and it hands you the nine regressions to read, which is where the actual insight lives.

If wins and losses are roughly balanced and the mean moved, the mean moved because a few questions changed a lot. Go read those, don’t ship the number.

Bootstrap for an interval

To attach uncertainty to a mean without assuming a distribution, resample your per-question scores with replacement, recompute the metric, and take percentiles of the resulting spread.

import random

def bootstrap_ci(scores, iters=2000, lo=2.5, hi=97.5):
    n = len(scores)
    means = []
    for _ in range(iters):
        sample = [scores[random.randrange(n)] for _ in range(n)]
        means.append(sum(sample) / n)
    means.sort()
    return (means[int(lo / 100 * iters)], means[int(hi / 100 * iters)])

For a paired comparison, bootstrap the per-question differences rather than the two means. If the resulting interval for the mean difference straddles zero, you can’t distinguish the change from noise on this set.

This handles the metrics where a closed-form standard error is awkward — nDCG, mean faithfulness over variable claim counts, judge-derived scores — which is most of them.

Multiple comparisons

If you try twenty configurations and report the best, you have selected on noise. With twenty draws, one will look good by chance even if all twenty are identical, and its measured advantage will be biased upward. The winner’s score is not an estimate of its true performance.

Three practices that keep this honest:

  • Confirm the winner on a fresh run. Re-run the chosen configuration and the baseline again, ideally with a re-sampled or held-out portion of the set. Effects that were noise shrink.
  • Count your comparisons out loud. “We swept eight k values and three thresholds” is 24 comparisons and should be stated when the result is reported.
  • Keep a holdout you rarely touch. Tuning against one set until the number rises is the failure mode described in when your eval set becomes training data.

Reduce the noise before you fight it

Cheaper than a bigger eval set, in rough order of leverage:

Freeze everything you can. Same index snapshot, same corpus version, same model version and parameters, same k, same prompt version. Record all of it with the result. Half of apparent noise is an unrecorded configuration difference.

Cache generations and judge verdicts keyed by a hash of the exact inputs. Re-running an unchanged case then returns the same value instead of re-rolling, which removes model non-determinism from every run where nothing relevant changed. It also cuts cost enough to let you run more often.

Judge once per claim, with a single-token output. Narrow tasks flip less.

Majority-vote the judge on high-stakes runs. Three calls, take the mode. Reduces flip rate at triple the cost; reserve it for release decisions.

Grow the set where it’s thin. Variance is per-slice, and slice-level conclusions need slice-level n — the floor discussion is in the aggregate score hides the bug.

What size set do you need

The honest answer is that it depends on the effect you want to detect and the metric’s variance, and you can measure both. Run your baseline twice, compute the per-question differences, bootstrap them, and read off the interval width. That’s your resolution: changes smaller than it are invisible on this set, today.

If the interval is wider than the improvements you’re likely to make, you have three options — a larger set, a paired design (usually enough on its own), or accepting that you can only detect large changes and saying so when you report.

Rules of thumb worth holding loosely: below about 50 questions almost nothing is measurable; 100 detects large changes; 300–500 gives enough resolution to trust modest ones. Those are starting points for how many questions to write, not substitutes for measuring your own variance.

The sentence to write in the report

Not “recall improved to 0.84”. Instead: “recall@5 improved on 22 of 150 questions and regressed on 9; bootstrapped 95% interval on the paired difference is +0.01 to +0.05; the nine regressions are all multi-source questions.”

That version is defensible, it survives a re-run, and it contains the next task.

metricsevaluation