A/B Testing a Retrieval Change

An A/B test answers the question offline evaluation can’t: did users get more out of the system. The difficulty is that RAG has no natural conversion event — nobody clicks “this answered my question” — so the primary metric has to be assembled from behavioural proxies, each of which is weak on its own.

Get the assignment unit and the guardrails right first. They’re the parts that make a test valid, and they’re cheaper to fix than a bad metric.

Assign by user or session, never by query

Random assignment per query breaks the experience: a user asks a follow-up and lands in the other arm, with a different retrieval configuration and no continuity. Behavioural metrics based on follow-up behaviour then measure your assignment scheme rather than your change.

Assign per user where you have identity, per session otherwise. Hash the identifier so assignment is deterministic and sticky across requests, and log the arm with every request — including in your traces, so a bad answer someone reports can be attributed to the right arm afterwards.

Guardrails before primary metrics

Guardrails are the metrics that stop a test regardless of whether the primary metric is winning. Decide them in advance, with thresholds:

• Latency, at the tail rather than the mean. Rerankers and verification passes land here.
• Error and timeout rate.
• Refusal rate, in both directions. A configuration that answers far more questions may just be answering ones it shouldn’t — measuring whether your system knows when to refuse.
• Empty or low-confidence retrieval rate.
• Cost per answer, if the arm adds model calls.

A test that wins on engagement while doubling tail latency and halving refusals is not a win, and without pre-declared guardrails that’s a conversation you’ll have after shipping instead of before.

Behavioural proxies, and what each one lies about

No single one of these is a success metric. Pick two or three, know their failure modes, and read them together.

Query reformulation rate. The share of sessions where the user rephrases and asks again. A decent negative signal — reformulation usually means the answer missed. It confounds with genuine multi-question sessions, and with users who reformulate because the system taught them to.

Follow-up depth. More turns can mean engagement or confusion. Ambiguous in isolation; useful when paired with reformulation, since reformulation-heavy depth reads differently from topic-shifting depth.

Escalation rate. The share of sessions that end in a human handoff, a support ticket, or a search of the underlying documents. Usually the best proxy available, because it’s a real cost the user chose to pay. Slow to accumulate and only available if your product has an escalation path.

Copy or citation-click events. A user copying the answer or opening a cited source is weak evidence the answer was worth using. Confounds with distrust — clicking a citation can mean “let me verify this because I don’t believe it”.

Session abandonment. Asking one question and leaving can mean satisfied or disgusted. Nearly uninterpretable alone; interpretable when the same arm also shows fewer reformulations.

Explicit feedback. Available and heavily biased. Fine as a sampler and poor as a metric — thumbs down is a weak label.

Task completion, if your product has one — the ticket got resolved, the form got filled, the order got placed. The only proxy that isn’t a proxy. If you have it, use it and use less of everything else.

Declare the primary metric before the test starts. Choosing afterwards from six proxies guarantees a winner and means nothing: with that many candidates, something moves favourably by chance.

Size and duration

Two rules do most of the work.

Run at least one full weekly cycle, ideally two. Query mixes differ on weekdays, weekends, and month boundaries, and a three-day test measures three days.

Compute the detectable effect in advance from your baseline rate and traffic, and if the honest answer is that you cannot detect a plausible effect, don’t run the test — you’ll get a null result and read it as “no difference” when it’s “no power”. For a rate metric, the standard error of a proportion, sqrt(p * (1 - p) / n), gives you the scale; the same reasoning as offline in was that a real improvement or noise?.

Two habits that protect the result: don’t stop early because the numbers look good — repeatedly checking and stopping on a favourable reading inflates false positives — and don’t slice the results afterwards looking for a segment that won. Pre-declare the segments you’ll examine, as with offline slices (the aggregate score hides the bug).

Interleaving, when you’re comparing rankings

For a pure ranking change — new reranker, different fusion, different k ordering — interleaving is more sensitive than an A/B test. Instead of splitting users, you merge both rankers’ results into one list per query, attribute each item to the ranker that contributed it, and count which ranker’s items get engaged with.

Every user sees a blend, so between-user variance disappears and you need far less traffic. Two conditions: it needs per-item engagement signals, which for RAG means visible source lists that users click, and it only compares rankings. It cannot evaluate a change to the generated answer.

When you don’t have the traffic

Most internal RAG deployments have dozens of users, not thousands. Online testing is unavailable and saying so is better than running an underpowered test and reporting the result.

What to do instead, in order of strength:

1. Blind paired human review on replayed live queries. Sample recent production queries, generate both arms’ answers, strip the labels, randomise order, and have reviewers pick the better answer with a reason. This is genuinely strong evidence — real queries, real comparison, controlled reviewer — and it needs a hundred queries rather than a hundred thousand sessions. The setup is in writing a rubric two reviewers agree on.
2. A shadow run. Serve arm A, compute arm B’s retrieval in the background, and diff. Zero user risk, and it tells you how often the arms differ at all — often the answer is “rarely”, which ends the discussion cheaply.
3. A staged rollout with monitoring. Ship to 10% and watch the guardrails and the unlabelled production signals rather than a primary metric — monitoring RAG quality in production.
4. A structured trial. Ten users, a fixed set of real tasks, before-and-after, with observation notes. Qualitative, small, and more informative than an underpowered A/B test that will return a null result either way.

Triangulate, don’t pick a winner from one number

The defensible claim comes from three sources agreeing: offline metrics improved on the relevant slices, blind human review preferred the new arm, and online guardrails held with the primary proxy moving in the right direction.

When they disagree, the disagreement is the finding. Offline up and online flat usually means the harness and the pipeline differ, or your set doesn’t match live traffic — why offline gains vanish online. Human review up and behavioural proxies flat often means the proxy is too noisy to detect a real but modest improvement. Both are worth more than a single green number.