Reproducing a Bad Answer Somebody Sent You
Someone pastes a screenshot into a channel. The assistant told a customer that damaged items can be returned within 30 days, and the policy says 14. No timestamp, no user, no link — just the two lines of text and a question mark.
You type the question into the assistant yourself and get the right answer. Nothing is fixed, and now you have two observations that disagree.
Reproduction is the step everyone skips, and it is where most of the diagnostic value is. The bisection in the five places a RAG pipeline breaks assumes you have a case that fails on demand. Getting there is its own procedure.
Why retyping the question isn’t reproduction
At least six things differ between the reported answer and your retry:
- The index. Someone may have re-ingested the corpus since. If the stale policy document was deleted this morning, the bug is now unreproducible and still shipped.
- The identity. Filters, tenant, locale, and permissions are inputs to retrieval. Your account probably sees a different corpus than the customer’s.
- The session. Prior turns, a rewritten query, or a summarised history that you don’t have.
- The model. A hosted version can move under you with no deploy — detecting regressions when the model updates.
- Sampling. Nonzero temperature means the same prompt has a distribution of answers, not an answer.
- Which path ran. Timeouts, fallbacks, and cache hits produce answers by a different route than the happy path.
Your retry tested a different system. That’s why it passed.
Get an identifier before anything else
The reply to the screenshot is one question: do you have the conversation, or a timestamp and the account? From either you can find the request record; from the screenshot alone you cannot.
What replay needs from that record is a short list — the exact query string as sent to retrieval (post-rewrite, if you rewrite), the identity and filters, the index or corpus version, the retrieved chunk IDs with scores, the assembled prompt, the model name and sampling parameters, and the response. If any of those aren’t recorded, this investigation is blocked and the fix is upstream of it.
Two of them are the ones teams most often lack: the index version and the post-rewrite query. Without the first you cannot tell a corpus problem from a retrieval problem; without the second you’re debugging a query the retriever never saw.
Three replay modes
Run them in this order. Each isolates a different layer, and the first one that fails tells you where to stop.
Mode 1 — replay the stored prompt. Send the recorded prompt verbatim to the recorded model with the recorded parameters. Retrieval is out of the picture entirely.
Mode 2 — replay retrieval on the recorded index. Same query, same filters, pinned corpus version. Compare the returned IDs to the recorded ones.
Mode 3 — replay the full pipeline on today’s system. The end-to-end retry, done properly: same query, same identity, current index.
| Mode 1 | Mode 2 | Mode 3 | Diagnosis |
|---|---|---|---|
| bad | — | — | Generation. The context was there and the model mishandled it. |
| good | IDs differ | — | The recorded retrieval isn’t reproducible: nondeterminism, or the index moved. |
| good | IDs match | bad | Assembly, truncation, or sampling — something between retrieval and the prompt. |
| good | IDs match | good | Fixed, environmental, or intermittent. Keep reading. |
Mode 1 failing is the good case: you have a deterministic, cheap, standalone reproduction that needs no infrastructure. Save that prompt as a fixture immediately.
Separate intermittent from fixed
If mode 1 passes and mode 3 passes, do not close the ticket. Run the full pipeline n times — five is usually enough to distinguish “never” from “sometimes” — and record the outcomes rather than the last one.
runs = [pipeline(query, identity=who) for _ in range(5)]
ids = {tuple(r.chunk_ids) for r in runs}
print(len(ids), "distinct retrieval sets")
print(sum(judge_bad(r.answer) for r in runs), "/", len(runs), "bad")Two distinct retrieval sets from an identical query means your retrieval is nondeterministic — approximate indexes break score ties differently across replicas or after a rebuild, and a chunk sitting at rank k can fall off the list on some runs. That’s a real finding, and it explains a category of complaint nobody can ever reproduce.
An intermittent failure is not a smaller bug than a deterministic one. It’s a rate, and a rate is measurable: run it fifty times overnight and you have a number to track instead of an anecdote.
When it genuinely won’t reproduce
Three explanations, in order of how often they turn out to be the answer.
The corpus changed. Diff the document that should have been retrieved between the recorded index version and now. If the stale version is gone, the bug was real and was fixed by an ingestion run — which means it will recur the next time a superseded document lingers. That’s a must_not_retrieve case, not a closed ticket: see regression testing a RAG pipeline.
A fallback path ran. Look for a trace with zero retrieved chunks, or a truncated context, or a retry. A retrieval timeout that lets the model answer from its own parameters produces exactly this shape: a fluent, confident, ungrounded answer that no amount of retrieval debugging will explain. Count these separately, because they are a reliability problem wearing a quality problem’s clothes.
Different treatment. An experiment arm, a cache entry, a canary deployment, a different prompt version for that account. The record should say which; if it doesn’t, that’s the field to add.
If none of the three explain it, write it down as unexplained with the evidence attached, and move on. An honest “cannot reproduce, here is what we ruled out” is worth more than a speculative fix that touches the prompt.
Where the case goes afterwards
Every reproduced failure ends up in two places: a code in the failure tally, so you learn what the distribution of complaints is rather than reacting to the loudest one (a failure taxonomy for RAG answers), and a permanent test case with the question, the identity, the expected retrieval, and the reason it exists.
That second artefact is the reason this procedure is worth the hour. One screenshot investigated properly yields a fixture that will still be catching the same class of regression a year later — which is more than can be said for the fix.