RAG Eval

Measuring a Prompt Change With Retrieval Held Still

You tighten the grounding instruction, run the end-to-end suite, and the judge’s average score moves up by three points (hypothetical figures). Retrieval also re-ran, returning slightly different chunks for a quarter of the questions, because the index has had two ingestion batches since your baseline. The three points are a mixture of your edit and that churn, and there is no way to separate them after the fact.

A prompt change affects exactly one stage. Everything upstream of that stage contributes variance and no signal, so the procedure is to remove it: freeze the contexts, then vary only the prompt.

Freeze the contexts as a fixture

For each question in your eval set, store what retrieval returned in one pinned run — IDs and the chunk text, because the text is what the prompt actually receives and re-fetching by ID will silently pick up an edited document.

{
  "id": "q_014",
  "question": "how long do I have to return a damaged item",
  "context": [
    {"id": "policy-returns-v3#sec-2", "text": "Damaged goods may be returned…"},
    {"id": "faq-shipping#damaged",   "text": "If your order arrives damaged…"}
  ],
  "corpus_version": "2026-06-30",
  "retriever": "hybrid-v2@k=5"
}

Now both prompt variants see byte-identical input. The comparison is paired at the question level, which is the arrangement that gets you the most statistical power for the least eval spend — see was that a real improvement or noise?.

Refresh the fixture deliberately, on its own schedule, and never in the same run as a prompt comparison. A stale fixture is a known limitation; a fixture that moved mid-experiment is a ruined experiment.

Include the awkward cases the fixture is most useful for: contexts with no relevant chunk (the refusal case), contexts with a superseded and a current document, and contexts where two chunks contradict each other. Those are where prompt wording changes behaviour most, and where a randomly-sampled fixture will have almost no examples.

Measure the deterministic things first

Before you spend a single judge call, run the assertions that need no model. A surprising share of prompt regressions are caught here:

  • Output parses. If you ask for JSON, it is valid JSON with the required keys.
  • Citation markers are present, well-formed, and resolve to IDs that were in the context.
  • No banned phrasing — the must_not_say list from scoring answers against a reference.
  • Length stays inside the range the interface can display.
  • The rendered prompt itself matches its snapshot for a fixed input, so an accidental whitespace or ordering change is visible as a diff rather than as a mysterious quality movement.

These are cheap, deterministic, and they fail loudly. Judge-based scores go second.

Then the four scores that can move

With retrieval fixed, a prompt edit can only change what the model did with the same material. Four measurements cover it, and they must be kept apart:

Faithfulness, claim by claim against the frozen context — the primary metric for any grounding-instruction change, and the one where the frozen fixture helps most, because claim verification against a moving context is barely comparable run to run. Method as in measuring faithfulness, not just answer quality.

Key-point coverage against reference answers. Catches the common trade: an instruction that suppresses unsupported statements also suppresses supported ones, and faithfulness rises while usefulness falls.

Abstention, on both subsets. Refusal rate on the no-relevant-chunk cases should go up or hold; refusal rate on the answerable cases must not. A prompt that refuses more is trivially more faithful, and this pair of numbers is the only thing that distinguishes caution from uselessness. Both belong in the report — measuring whether your system knows when to refuse.

Format and tone conformance against the rubric, if the change was stylistic.

The judge’s style bias is the main trap

Prompt changes are the single worst case for LLM-as-judge, because they change surface features the judge is known to respond to. A variant that produces longer, better-structured, more confident answers tends to score higher on a holistic 1–5 scale without any change in correctness, and that is precisely the kind of edit you make when you’re editing prompts.

Three defences, all of which you want:

  1. Claim-level checks over holistic scores. “Is this sentence supported by this chunk” is a far less style-sensitive question than “rate this answer.”
  2. Blind pairwise with position swap. Show the judge both answers without labels, then run it again with the order reversed. Disagreement between the two orderings is a position-bias measurement, not a tie — count it as no-preference.
  3. A human-scored sample of the disagreements, every time, because the judge is an instrument that has to be re-checked whenever the thing it measures changes shape: validating an LLM judge before you trust it.

Pin the model version for the comparison as well. A prompt evaluated against a model that shifted underneath it produces a delta belonging to neither.

What the frozen result licenses

It licenses a statement about generation: given these contexts, variant B was more faithful and no less complete, paired over N questions, with the judge validated against a human sample. That is a strong, honest claim, and it is the claim that lets you ship.

It does not license an end-to-end claim, for two reasons. Your fixture’s contexts came from one retriever at one k, so a prompt tuned on them can be tuned to their quirks. And a prompt change can alter what fits: if your edit lengthens the instructions enough to displace a chunk, or changes the assembly order, retrieval is no longer being held still and the result must be re-measured as a pipeline change rather than a prompt change.

So finish with one live end-to-end run before shipping, and expect a smaller effect than the frozen comparison showed. The frozen number tells you the mechanism works; the live number tells you how much of it survives contact with real retrieval. Keeping those two claims distinct is the whole point of component or end-to-end, and prompt work is where the distinction earns its keep most often.

evaluationfaithfulness