RAG Eval

Validating an LLM Judge Before You Trust It

An LLM judge is a measurement instrument, and an instrument nobody has calibrated produces numbers of unknown value. Before a judge score influences a decision, you need to know how often it agrees with a human, in which direction it errs, and on which kinds of input it falls apart.

None of this is expensive. It’s a few hundred labelled examples and an afternoon, and it converts “the judge says faithfulness is 0.88” into a statement you can defend.

Build a probe set, not a sample

The validation set for a judge is not a random sample of your eval set. Random sampling from a mostly-passing distribution gives you a hundred easy positives and four hard cases, and the hard cases are the entire question.

Construct it deliberately, with human labels on every item:

  • Clear passes and clear fails, in roughly equal numbers. Balance matters, because a judge that always says “supported” scores well against an unbalanced set.
  • True-but-unsupported claims. A claim that is factually correct and absent from the context. This is the single most important probe: a judge that uses its own world knowledge marks these as supported and silently destroys your faithfulness metric.
  • Contradicted claims, where the context says the opposite. Judges confuse “contradicted” with “not mentioned” more often than either with “supported”.
  • Numeric and quantitative claims. “Within 30 days” against a context saying 14. Small token differences, large semantic ones.
  • Negations. “X is not covered” against a context stating X is covered. A well-known weak spot.
  • Multi-part claims, where one half is supported and the other isn’t. The correct verdict is not supported, and leniency shows up here first.
  • Verbose but wrong answers, to probe length bias, and terse but right ones, to probe the same bias from the other side.

Fifty to a hundred well-chosen probes beat five hundred random ones. Label them with two humans and adjudicate the disagreements, because your labels are the reference and an unreliable reference makes the whole exercise circular — writing a rubric two reviewers agree on covers that process.

The four numbers to compute

Agreement rate. Fraction of probes where the judge matched the human label. The headline, and insufficient alone.

Per-class recall. Of the items humans labelled unsupported, what fraction did the judge catch? This is the number that matters, because unsupported claims are what you’re hunting and a judge can post high overall agreement while missing most of them.

Bias direction. Does it over-report supported or under-report it? A judge that is consistently lenient by a known margin is still a usable instrument — you read it the way you read a scale that’s 2 kg heavy. A judge whose errors are unpredictable is not.

Chance-corrected agreement. Cohen’s kappa or similar, for the same reason as with human reviewers: raw agreement on a skewed set flatters a judge that always answers the majority class.

Report all four with every judge-derived metric. In practice this means your eval output carries a header block: judge model, judge prompt version, probe-set version, agreement, and unsupported-class recall. Without it, a faithfulness number from March is not comparable to one from July.

Stability, which is not the same as accuracy

Set temperature to zero and you still won’t get identical outputs run to run — batching, hardware, and provider-side changes all introduce variation, and this is a property of hosted inference rather than a bug in your code.

Measure it: run the judge over the same 100 probes three times and count how many verdicts changed.

def flip_rate(runs):
    """runs: list of lists of verdicts, same order, same inputs."""
    n = len(runs[0])
    flips = sum(len({r[i] for r in runs}) > 1 for i in range(n))
    return flips / n

A non-trivial flip rate puts a floor under the smallest change your judge can detect. If 4% of verdicts move between identical runs, a 2% improvement in faithfulness is noise, and no amount of eval-set size fixes it. Combine that floor with sampling variance when you decide whether a delta is real — was that a real improvement or noise?

Mitigations, in order of cost: constrain the output to a single token, keep each call to one claim, run the judge multiple times and take a majority vote for high-stakes runs, and cache verdicts keyed by the exact input so a re-run of an unchanged case doesn’t re-roll the dice.

Design choices that improve agreement

One claim per call. Batching claims into one prompt introduces order effects and degrades accuracy on the later items.

Single-token verdicts. Free-text rationales are useful for debugging and should come after the verdict token or in a separate call. A judge that reasons at length before answering can talk itself into a different verdict, and you cannot parse a paragraph reliably.

Explicitly forbid outside knowledge in faithfulness prompts. This is the fix for the true-but-unsupported probe, and it is worth re-testing after every prompt edit.

Use a different model family from the generator. Judges show a preference for outputs resembling their own. Where you can’t — one provider, one model — say so when you report the number, and lean harder on human calibration.

Pairwise comparisons in both orders. When judging A against B, position bias favours whichever is shown first. Run every pair twice with the order swapped and treat disagreement between the two runs as a tie. Cheap, and it removes a bias large enough to reverse a conclusion.

Re-validate on these triggers

A validated judge does not stay validated. Re-run the probe set when:

  • The judge model version changes, including silent provider-side updates. This is the one people miss, and it changes your metric without touching your code — detecting regressions when the model updates.
  • The judge prompt changes at all. Judge scores are comparable within a prompt version, not across versions. Treat a prompt edit like a schema migration: bump the version, re-validate, annotate the break in your trend chart.
  • The answer distribution changes. A judge validated on short factual answers has unknown behaviour on the long structured answers your new prompt produces.
  • The domain changes. New corpus, new vocabulary, new failure modes.
  • On a schedule — quarterly is a reasonable default — because the first three happen without anyone noticing.

What a validated judge still cannot do

It cannot tell you whether the answer is true, only whether the context supports it. It cannot tell you whether the question was worth answering. And it cannot substitute for the human sample, because the judge’s agreement rate is measured against that sample and shrinking it shrinks your confidence in everything downstream.

The honest framing is that a validated judge lets you scale human judgement to thousands of cases with a known error rate. That is a large win and it is not the same as automating evaluation. Keep a standing human review slice; it’s what makes the judge’s numbers mean anything at all.

evaluationfaithfulness