RAG Eval

Detecting Regressions When the Model Updates

Every eval you run assumes the model is a fixed component. It isn’t. Hosted models get updated, aliases point at new versions, defaults change, and behaviour shifts without a line of your code changing — so your quality can move on a day when nobody deployed anything.

Two defences: record enough to notice, and run evaluation on a schedule rather than only on commits.

Why a code-triggered suite can’t see it

A CI-triggered suite runs when you change something. A model updated on a Tuesday when your team shipped nothing produces no run, so the first evidence is either a user complaint or a confusing eval result days later attributed to whatever change happened to land next.

The fix is a scheduled canary run: the same eval subset, the same pinned configuration, executed nightly whether or not anything changed. Its whole purpose is to attribute a movement to time rather than to a commit. It belongs in the cadence plan alongside everything else — which evals run on every commit.

Keep the canary small, stable and cheap so it can genuinely run every night: a fixed stratified subsample, a fixed seed, cached where caching is valid. Note the one exception to caching — a canary whose generations come from cache cannot detect a model change, which is the entire point. Key the cache on the resolved model version so a new version invalidates it automatically, and re-run a small uncached slice every night regardless.

Pin what you can, record what you can’t

Pin explicit versions rather than floating aliases wherever the provider offers them. An alias is a convenience that trades reproducibility for automatic upgrades, which is the wrong trade for the model under your evaluation.

Record the resolved version with every result. Not the alias you requested — the version string the API actually reports back. This one field converts “quality dropped last week and we don’t know why” into a one-line answer, and it costs nothing to log.

The full configuration block on every eval artefact:

{
  "generator": {"model": "…", "resolved_version": "…", "temperature": 0, "max_tokens": 800},
  "judge":     {"model": "…", "resolved_version": "…", "prompt_version": "faith-v4"},
  "embedder":  {"model": "…", "resolved_version": "…", "dim": 1024},
  "prompt_version": "answer-v11",
  "corpus_version": "2026-07-18",
  "chunking_version": "c7",
  "eval_set_version": "v9",
  "k": 5
}

If a metric moves and every field is identical to yesterday, the change came from outside your system. If a field changed, you have your suspect. This is the same argument as pinning the corpus in regression testing a RAG pipeline, extended to components you don’t own.

Your judge drifts too, and that’s worse

The subtle case: the judge model updates. Nothing about your system changed, and your faithfulness number moves. Chase it and you’ll spend a week debugging retrieval that is working fine.

Guards, in order of importance:

  1. Pin the judge version harder than the generator version. A measurement instrument that changes silently is worse than a slightly older one.
  2. Keep a labelled probe set and re-run it whenever the judge version changes, scheduled as well as on change. If agreement with human labels moved, your metric’s meaning moved — validating an LLM judge before you trust it.
  3. Annotate the break. When you do move judge versions, mark the date in the metric history. Scores are comparable within a judge version, not across.
  4. Include judge agreement in the canary output, not just the faithfulness score. A dashboard showing only the derived metric cannot distinguish a system regression from an instrument change.

What to look at when you upgrade deliberately

When you choose to move to a new model version, the useful comparison is paired: both versions, same questions, same retrieved context, same prompt, at the same time. Read wins and losses rather than means — was that a real improvement or noise?.

Then check the dimensions that move most across model versions, which are usually not accuracy:

Instruction following. Formatting rules, citation formats, “answer only from the context”. A new version may follow instructions more literally or less, and downstream parsers break on format changes rather than on quality changes. Your prompt-snapshot and output-schema tests are the cheap detector here.

Refusal propensity. Providers tune this between versions, and it moves both cells of the abstention matrix at once. Re-measure over-answering and over-refusal together, not just one — measuring whether your system knows when to refuse.

Verbosity. Longer answers change latency, cost, and any judge-derived quality score that has a length bias. Track mean answer length as a monitored metric; it’s a sensitive, free indicator that something changed.

Context utilisation. How reliably material deep in a long context gets used varies between versions. If your prompts are long, re-run a position-sensitivity check: same context in shuffled order, and see whether answers change more than they used to.

Prompt sensitivity. Expect your prompt to need retuning. A prompt refined against one version is fitted to it, and some of its wording is compensating for that version’s quirks. Budget for a retuning pass rather than treating a small drop as a verdict on the new model.

Expect the upgrade to be a trade

Model changes are rarely uniform improvements on a specific corpus. The pattern to plan for is better on some slice, worse on another, with the aggregate roughly flat — which means the decision comes from per-slice results and from which slice matters to your users, not from the headline number (the aggregate score hides the bug).

Keep the ability to roll back until you’ve seen the online numbers, since offline agreement doesn’t guarantee the online ones follow — why offline gains vanish online.

The minimum viable version of all this

If you do nothing else, do these three:

  1. Log the resolved model versions — generator, judge, embedder — with every eval result and every production request.
  2. Run a small fixed eval subset nightly, on a timer, uncached, and store the results with the configuration block.
  3. Alert on movement in that series with no configuration change.

That’s a scheduled job, a JSON file per night, and a threshold. It converts an entire category of unexplained quality drift into a dated event with a probable cause, which is most of the work of debugging it.

evaluationdebugging