RAG Eval

Why Offline Gains Vanish Online

Offline evaluation improved, you shipped, and nothing changed for users. This is common enough to be the default expectation rather than a surprise, and every cause is findable.

The gap has two kinds of source: your eval set describes a different population than production does, and your eval harness runs a different pipeline than production does. The second kind is more common and much easier to fix.

The harness is not the pipeline

Check this first, because it’s mechanical and it’s usually where the gap lives.

Different parameters. The harness retrieves k=10, production serves k=5. The harness has no context-length cap; production truncates. The harness uses temperature 0; production uses 0.7. Every one of these makes offline numbers describe a system nobody is running.

Different retrieval path. Production applies permission filters, tenant scoping, a metadata pre-filter, or a query rewriting step the harness skips. Filtered retrieval is a different retrieval problem — recall measured unfiltered is an upper bound on the filtered case, sometimes a loose one.

Different index. Offline evaluates a freshly built snapshot; production serves an index that’s been incrementally updated for months, with deletions that left tombstones and documents that were re-embedded with a different model version mid-migration.

Different prompt. The prompt template in the harness is a copy that drifted from the one in the serving code. If your prompt lives in two places, they differ; this is not a hypothetical.

Different context assembly. Ordering, deduplication, metadata headers, and truncation rules are code, and if the harness reimplements them, it reimplements them slightly differently.

The fix is structural: evaluate through the production code path. Call the same function production calls, with the same configuration object, and let the harness supply only the question and collect the output. Anything the harness reimplements is a place the two can diverge silently. If that’s impossible today, make the parameter block a single shared artefact and assert equality in a test — the prompt-snapshot idea from regression testing a RAG pipeline applied to the whole config.

The population is not the same

Then the harder half.

Query distribution. Your eval questions were written or sampled at some point in the past; live queries are shorter, messier, more repetitive, and more concentrated on a few topics than any hand-built set. Improvements on the long tail of your set can be irrelevant if 40% of production traffic is five questions.

No session context. Production queries arrive mid-conversation, full of pronouns and ellipsis — “what about international?” — and depend on a rewriting step your single-turn eval set never exercises. If you have multi-turn traffic and a single-turn eval set, an entire failure class is unmeasured.

Users adapt. People learn what the system answers well and stop asking what it answers badly. Your production distribution is partly a product of your system’s past quality, which means it shifts after you ship and the before/after comparison isn’t measuring a fixed population.

Stale labels. Offline improvements measured against labels that no longer match the corpus are improvements against history — a re-chunk invalidates your eval set.

Your metric isn’t their preference. Faithfulness and relevance are proxies for usefulness. A change that raises faithfulness by making answers more hedged and more citation-heavy can lower user satisfaction. Judge-derived metrics are especially prone to this, since tuning against a judge fits the judge’s tastes — when your eval set becomes training data.

Latency is part of quality online and absent offline. A change that adds a reranker and a verification pass improves every offline number and slows the answer. Offline evaluation has no term for that.

Shrink the gap: replay live queries

The most effective single practice: sample real production queries, replay them offline, and score them.

{
  "id": "replay_2026_07_21_0413",
  "question": "and for international orders?",
  "prior_turns": ["how long do I have to return a damaged item"],
  "tenant": "acme",
  "filters": {"locale": "en-GB"},
  "served_answer": "…",
  "retrieved_ids": ["…"],
  "sampled_from": "production"
}

Capturing the session context, the filters, and what production actually retrieved gets you three things at once: a query set with the right distribution, the ability to diff your offline retrieval against what production really returned, and a stream of new eval cases that needs no invention.

Sample it continuously and refresh a rolling “recent traffic” slice in your eval set. Report that slice separately from the curated set — the curated set measures the cases you care about, the replay slice measures the cases you have.

Shrink the gap: measure the same thing in both places

Pick metrics you can compute offline and online, and watch them in both:

  • Refusal rate. Computable anywhere, sensitive to almost every change.
  • Answer length distribution. Free, and it moves when prompts or models move.
  • Retrieval score distribution for the top result. Detects corpus and query drift.
  • Faithfulness on a sample. Same judge, same prompt version, both offline and on sampled live traffic. If offline faithfulness rose and the online sample didn’t, the gap is real and now it’s quantified.

Divergence between the same metric measured in two places is the clearest evidence you can get that the harness and the pipeline disagree, and it’s a much sharper signal than comparing an offline score against a user-satisfaction metric that moves for a dozen reasons. The production side of this is monitoring RAG quality in production.

Then confirm online

Offline evaluation’s job is to select candidates and prevent regressions, not to prove user impact. The final confirmation is an online comparison with behavioural metrics — A/B testing a retrieval change — and where traffic is too small for that, a blind human review on replayed live queries is the honest substitute.

The expectation to set

Say it before you ship, not after: offline improvements are necessary and not sufficient, they attenuate online, and occasionally they reverse. A team that expects attenuation investigates the size of the gap. A team that expects transfer concludes that evaluation is useless, stops doing it, and ships on anecdote — which is the actual failure mode this whole discipline exists to prevent.

evaluationmetrics