RAG Eval

Monitoring RAG Quality in Production

Offline evaluation runs against a frozen set and therefore cannot see the things that break in production: the corpus drifting, the query mix shifting, a model updating under you, a config change that only exists in the serving path. Production has all of those and no labels.

What it does have is distributions, and a distribution that moves is a signal. Monitor shape rather than correctness, and add a small set of labelled canaries for the cases where you need a real quality number.

This is not infrastructure monitoring

Latency, error rate, index size and memory pressure belong on your infrastructure dashboard and they’re not what this post is about. A vector store can be perfectly healthy while answer quality collapses, and the two dashboards answer different questions.

The signals below are about what the system is saying, and every one of them can move while every infrastructure metric stays flat.

Eight unlabelled signals

1. Top-1 retrieval score distribution. Log the best similarity score for every query and watch the distribution, not the mean. It shifts when the corpus changes, when the query mix changes, and when an embedding migration is half-finished. Absolute values are meaningless across models; the shape over time is the signal.

2. Weak-retrieval rate. The fraction of queries whose top result falls below whatever threshold you consider marginal. A single interpretable number, and a leading indicator: it rises before complaints do.

3. Refusal rate. Sensitive to nearly every change in the system — prompt, model, corpus, thresholds. A step change with no deploy means something outside your code moved. Classify refusals properly rather than string-matching, because hedged answers are not refusals (measuring whether your system knows when to refuse).

4. Answer length distribution. Free to compute and surprisingly informative. Model updates, prompt edits and context-truncation bugs all show up here first (detecting regressions when the model updates).

5. Query novelty. The share of live queries unlike anything in your eval set — measurable as distance to the nearest eval question in embedding space. When this climbs, your offline numbers are describing an increasingly different population, and it’s the trigger to refresh the set from logs.

6. Retrieval concentration. Which documents get retrieved, and how often. A few documents dominating every response usually means duplicates, an over-broad chunk, or a document whose embedding sits suspiciously central. Documents never retrieved at all are the other tail: either dead weight or unreachable content.

7. Reformulation rate. The share of sessions where a user immediately rephrases. The best cheap behavioural proxy for “that answer missed”, with the caveats in A/B testing a retrieval change.

8. Escalation rate. Sessions ending in a human handoff or a support ticket. Slow, noisy, and the closest thing to ground truth you get for free.

None of these measure correctness. Together they detect change, and in production, detecting change fast is most of the job.

Sampled judging for an actual quality number

For a real quality figure on live traffic, sample and judge.

Take a small random sample of production answers per hour — enough to accumulate a usable daily number, few enough to fit a fixed budget — and run claim-level faithfulness and relevance scoring on them with the same judge and prompt version as your offline suite.

{
  "sampled_at": "2026-07-27T14:03:11Z",
  "arm": "control",
  "judge": {"model": "…", "resolved_version": "…", "prompt_version": "faith-v4"},
  "faithfulness": 0.83,
  "relevance": 1.0,
  "refused": false,
  "top1_score": 0.71,
  "n_chunks": 5
}

Three rules keep it honest. Use the same judge version as offline, or the two numbers aren’t comparable — and divergence between them is one of the most useful things you can measure (why offline gains vanish online). Sample uniformly rather than from complaints, or you’re monitoring your worst case only. And cap the spend explicitly, because a sampled judge on live traffic is a cost that scales with your success.

Canary queries, run on a schedule

The cheapest high-value monitor: twenty to fifty questions with known good answers, fired at the real production endpoint every few minutes, scored automatically against expected retrieved documents and expected key points.

This is the only thing on this list that catches config-level breakage within minutes: an index that didn’t get rebuilt, a filter deployed too broadly, a permission change that silently excludes a document class, an embedding version mismatch between indexing and querying. Every one of those looks fine on an infrastructure dashboard and produces obviously wrong answers.

Keep canaries deliberately narrow — exact expected document IDs, exact expected facts — because a canary that fails ambiguously gets muted. And route them through the real serving path, including auth and filters, since bypassing the path defeats the purpose.

Alert on shape, not on quality

Absolute thresholds on quality signals produce false alarms, because these distributions move for benign reasons all the time. Alert on movement instead:

  • Week-over-week distribution shift on the score and length distributions, compared against the same weekday to control for the weekly cycle.
  • Step changes with no deploy. The most informative alert you can build, because it points immediately outside your code — corpus, model, or traffic.
  • Canary failures, immediately and loudly. These are binary and near-zero false positive.
  • Sampled judge scores below a floor set from your own history, evaluated on a daily aggregate rather than hourly noise.

Attach the configuration block to every data point — resolved model versions, prompt version, corpus version, index version. Without it, “quality dropped on the 14th” is a mystery; with it, it’s usually a one-line answer.

The dashboard that gets read

Four panels, in this order:

  1. Canary pass rate — is the system fundamentally working right now.
  2. Weak-retrieval rate and refusal rate — the leading indicators, on the same time axis.
  3. Sampled faithfulness and relevance, daily, with the offline number from the nightly run drawn on the same chart. The gap between the two lines is the health of your entire evaluation apparatus.
  4. Query novelty and retrieval concentration — is the population you’re serving still the one you evaluate against.

Then close the loop: every production case that looks wrong gets a trace pulled and a code assigned, and the codes get counted. That’s a failure taxonomy for RAG answers, and production monitoring is where its input comes from. Monitoring that produces alerts but never produces eval cases decays into a wall of graphs nobody reads.

evaluationdebugging