RAG Eval

Scoring Answers Against a Reference

If your eval set includes a written reference answer, you can score correctness automatically rather than paying a human every run. The catch is that the obvious ways of comparing two texts — string overlap, embedding similarity — measure the wrong thing on anything longer than a phrase.

What works is decomposing the reference into the key points it contains and checking coverage of each. That survives paraphrase, catches omission, and produces a number you can act on.

Why string metrics stop working

Exact match is fine and genuinely reliable for extractive short answers: a date, an amount, a policy duration, a part number. If your eval set is 200 questions whose answers are single values, exact match after light normalisation is the correct metric and you should use it. Cheap, deterministic, no judge to validate.

It collapses the moment answers are sentences. “You have 30 days from delivery” and “Returns are accepted within thirty days of the delivery date” are the same answer and share almost no tokens.

Token overlap metrics (F1 over words, or the n-gram overlap scores borrowed from machine translation and summarisation) partially patch this and introduce a worse property: they reward length and vocabulary imitation. An answer that copies the reference’s phrasing while inverting its meaning scores well. Negation is nearly invisible to them, and negation is exactly where policy answers go wrong.

Embedding similarity between answer and reference is the modern default and it fails in a specific, dangerous direction: it scores topical proximity. An answer that contradicts the reference on one detail while discussing the same subject in the same register lands very close in embedding space. Two texts about the return window score high whether they agree or not.

That’s not a small caveat. The whole point of correctness scoring is to catch the wrong detail, and this metric is structurally blind to it.

Key-point coverage

Write the reference not as prose but as a list of the facts the answer must contain.

{
  "id": "q_014",
  "question": "how long do I have to return a damaged item",
  "key_points": [
    "Damaged items can be returned within 30 days of delivery",
    "Photographic evidence of the damage is required",
    "Return shipping is paid by the company for damaged items"
  ],
  "must_not_say": [
    "the standard 14-day window applies to damaged items"
  ]
}

Scoring is then two independent checks:

Coverage = key points present in the answer / total key points. Catches omission, which is the failure mode long-form answers actually have.

Violations = count of must_not_say items the answer asserts. Catches the specific wrong answers you’ve seen in the wild. This field is worth more than it looks: it turns each production failure you investigate into a permanent, precisely-targeted regression check.

def score_reference(answer, item, entails):
    """entails(text, claim) -> True if text asserts claim."""
    covered = sum(entails(answer, kp) for kp in item["key_points"])
    violated = sum(entails(answer, bad) for bad in item.get("must_not_say", []))
    return {
        "coverage": covered / len(item["key_points"]),
        "violations": violated,
    }

entails is a judge call — one claim at a time, same discipline as claim-level faithfulness checking. It is a much easier task than holistic answer grading, because the judge compares one short sentence against one text and returns one token. Easier tasks mean better agreement with humans, which is the only reason to prefer this decomposition over asking a model “score this answer 1–5”.

Validate the judge on this task before trusting the numbers — validating an LLM judge before you trust it.

Coverage is not the whole score

Coverage says nothing about what else the answer contains. Two answers with identical coverage can differ enormously: one is three sentences, the other buries the same three facts in eight paragraphs of adjacent policy.

So report coverage alongside two things it can’t see:

Coverage plus violations plus faithfulness plus relevance is four numbers, and four numbers is the right amount for a generation dashboard. One blended score would be easier to read and would tell you nothing about what to do.

Where reference answers come from, and go stale

Write them from the corpus, not from memory. A reference answer written by someone who knows the domain but hasn’t checked the documents encodes what the domain expert believes, not what the system is supposed to say. When they diverge, you’ll spend a day debugging retrieval for a question whose reference is wrong.

Record the source. Each key point gets a document ID, ideally the same IDs as the retrieval labels. This lets you distinguish “the answer is wrong” from “the corpus changed and the reference is stale”, which otherwise looks identical in the score.

Re-verify on corpus updates. A reference answer is a claim about your corpus at a point in time. When the policy changes, the reference is wrong and the system is right, and the eval run will confidently report a regression. Version the reference set with the corpus snapshot, the same discipline as retrieval labels — see building a retrieval eval set.

Keep the unanswerable cases reference-free. For questions the corpus can’t answer, the correct output is a refusal, and coverage is undefined. Score those with the abstention machinery instead — measuring whether your system knows when to refuse.

When not to bother

Reference-based scoring is worth its cost when answers are short-to-medium, factual, and stable. It’s a poor fit for three cases.

Genuinely open-ended questions — “summarise the risks in this contract” — where many good answers exist and no key-point list captures them. Use a rubric with human review on a sample instead.

Highly volatile corpora, where reference answers rot faster than you can maintain them. Fall back to faithfulness and relevance, which are reference-free by construction and score against whatever the corpus says today.

Very small eval sets. If you have 30 questions, human review of all 30 is cheaper than building and maintaining reference answers, and better. Reference scoring pays off at the point where you’re running the set often enough that human review can’t keep up.

evaluationmetrics