RAG Eval

Writing a Rubric Two Reviewers Agree On

Human review is the calibration layer under every automated metric you have, and most of it is wasted because the rubric is too vague to produce repeatable scores. If two competent reviewers reading the same answer disagree, the number they produce cannot detect a change in your system.

The fix is not more reviewer training. It’s a rubric with fewer dimensions, coarser scales, and concrete anchors.

Start by cutting dimensions

The instinct is to score everything: accuracy, completeness, groundedness, tone, structure, conciseness, helpfulness. Seven dimensions on a 1–5 scale is 35 judgements per answer, reviewers fatigue after a dozen answers, and the dimensions correlate so heavily that the seven numbers carry maybe two numbers’ worth of information.

Score three things, at most four. For a RAG assistant the durable set is:

Correct — are the factual claims right, as far as the reviewer can verify from the cited sources? Complete — does it address everything the question asked? Grounded — is every claim traceable to the provided sources?

Tone and structure go in a free-text comment field. They matter for the product and they don’t need a score, because nobody ships a change based on a 0.1 movement in tone.

Use binary or three-point scales

A 1–5 scale invites reviewers to express confidence rather than judgement, and different reviewers use the range differently — one person’s 3 is another’s 4, permanently. The spread you measure is partly your system and partly reviewer personality, and you can’t separate them.

Binary is better where the question genuinely has two answers. “Is every claim supported? yes/no.” Three points work where partial credit is real: fails / partially meets / meets.

If you must have a five-point scale for a stakeholder who wants an average, define all five points as behaviours and accept that you’ll mostly see 2s and 4s.

Anchor every level with an example

This is the highest-leverage part of a rubric and the part usually missing. Each level of each dimension gets a one-line definition and a real example answer from your own system.

Complete

  • Meets — addresses every part of the question. Example: Q asks “can I expense this and who approves it”; answer covers both.
  • Partial — addresses the main part, omits a secondary part, or omits an important condition. Example: same Q, answer covers expensing only.
  • Fails — does not address what was asked, or is a refusal where the sources contain the answer.

Anchors written from your own failures do two things: they make the scale mean the same thing to everyone, and they force you to articulate the failure modes you actually have. Refresh them when a new failure mode shows up — the codebook from your failure taxonomy is the natural source.

Make the reviewer’s job small

Show the sources. A reviewer scoring groundedness without the retrieved context is guessing. Show the exact chunks the model received, in order.

Blind the arm. When comparing two systems, don’t label which answer came from which. Reviewers who know which one is “the new one” find it better. Randomise the order of presentation too, and record the order so you can check for a position effect afterwards.

Require evidence for a fail. A reviewer marking “not grounded” must paste the unsupported claim. This costs seconds, kills a large fraction of careless scores, and hands the engineer a ready-made bug report.

Cap the session. Reviewer quality degrades within an hour. Twenty to thirty answers per sitting, and shuffle so no reviewer gets all the hard cases.

Measure agreement, then fix the rubric

Before any rubric output goes into a decision, have two reviewers independently score the same batch — 30 to 50 answers is usually enough to find the problems.

Two things to compute:

Raw agreement — the fraction of items where both gave the same score. Intuitive, and inflated when one score dominates: if 90% of answers are “meets”, two reviewers who always say “meets” agree 90% of the time while measuring nothing.

A chance-corrected coefficient — Cohen’s kappa for two reviewers, Krippendorff’s alpha if you have more or ordinal scales. These subtract the agreement you’d expect from the marginal distributions alone, which is exactly the inflation above. Conventional rules of thumb treat kappa in the 0.6–0.8 band as usable and below 0.4 as weak; those bands are convention rather than law, and the honest use of the number is comparative — did the rubric revision improve agreement on your data?

def cohens_kappa(a, b, labels):
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    expected = sum(
        (a.count(l) / n) * (b.count(l) / n) for l in labels
    )
    return (observed - expected) / (1 - expected)

Then do the part that matters: read the disagreements. Every disagreement is either a genuinely ambiguous answer or an underspecified rubric, and the second kind is fixable. Adjudicate as a group, decide what the rule should be, write it into the anchor, and re-run. Two or three rounds of this is normal, and the rubric that comes out of it is the asset — not the scores from round one.

What the agreement number is for

Reviewer agreement is the ceiling on every automated metric calibrated against these labels. If two humans agree 70% of the time on groundedness, an LLM judge agreeing 70% with one of them is performing at human level, and chasing 90% agreement is chasing a target that doesn’t exist.

Report it every time you report a judge’s agreement rate. A judge at 82% agreement sounds mediocre next to nothing and excellent next to a human ceiling of 84%. The comparison is the only way to read it — see validating an LLM judge before you trust it.

Sampling: what gets reviewed

You cannot review everything, and reviewing a random sample of production traffic spends most of the budget on easy successes.

A workable split for a weekly review budget:

  • A stratified random sample of production answers. The baseline, and the only unbiased part. Stratify by question type or tenant so small segments appear at all.
  • Low-confidence cases — answers where retrieval scores were weak or the judge was uncertain. Highest density of real problems.
  • Disagreement cases — where two automated metrics conflict, e.g. high faithfulness with low relevance. These are where your metrics are wrong, which is more valuable to know than one more bad answer.
  • A fixed regression slice — the same 20 answers to the same 20 questions every week. Human-scored trend over time, immune to sampling noise.

Keep the proportions fixed, or your week-over-week numbers move because the sampling changed. That’s the same trap as an unstable eval set, and it’s covered in when your eval set becomes training data.

evaluation