RAG Eval

A Failure Taxonomy for RAG Answers

Metrics tell you how much is broken. They don’t tell you what is broken, and the gap between those two is where most RAG work stalls — a faithfulness score of 0.78 supports no decision about what to do on Monday.

Error analysis closes it. Sit down with fifty failures, assign each a stage and a failure type, and count. The output is a ranked list of causes, which is a work queue.

Sample fifty failures properly

Fifty is the right order of magnitude. Enough to see which modes dominate, few enough to read carefully in a day. Twenty is enough to find the biggest one.

Stratify. Sampling only the worst-scoring cases finds the catastrophic modes and misses the common mediocre ones. A workable mix: half from low-scoring eval cases, a quarter from production cases with negative feedback or escalation, a quarter random from production. Record which stratum each came from so you can weight the counts afterwards.

Bring the trace. Coding a failure from the answer alone is guessing. You need, per case: the query, the retrieved chunk IDs with scores and ranks, the assembled prompt, the answer, and — critically — whether the answer exists in the corpus at all. Without that last item you cannot distinguish a retrieval failure from a corpus gap, and they get fixed by different teams.

If you don’t have traces yet, that’s the finding: stop and instrument, because error analysis without them takes ten times as long and produces guesses.

Code each failure with a stage and a type

Two fields, both required. The stage is where it broke; the type is how.

Stage — corpus, extraction, chunking, indexing, retrieval, reranking, prompt assembly, generation, or the question itself. The bisection procedure that determines it is in the five places a RAG pipeline breaks.

Type — a starter codebook, to be adapted:

Code Stage Signature
corpus-gap corpus the answer isn’t in the corpus; correct behaviour was a refusal
stale-doc corpus the corpus contains an outdated answer and reported it faithfully
extraction-loss extraction text was mangled or dropped — tables, columns, scans, headers
boundary-split chunking the answer was cut across two chunks and neither is sufficient
context-orphan chunking the chunk lost the heading or subject that made it interpretable
not-indexed indexing the document exists and isn’t in the index
vocab-mismatch retrieval the user’s words and the corpus’s words don’t meet
duplicate-crowding retrieval near-duplicates filled top-k, squeezing out other relevant material
version-confusion retrieval several versions retrieved, wrong one ranked first
filter-overreach retrieval a metadata or permission filter excluded the answer
rank-too-low reranking retrieved but below the cutoff the generator reads
truncated prompt assembly the relevant chunk was cut by a context limit
instruction-ignored generation the model broke a rule it was given
unsupported-claim generation asserted something the context doesn’t support
partial-answer generation answered one part of a multi-part question
over-refusal generation refused when the context contained the answer
ambiguous-question question genuinely underspecified; a clarification was the right output

Allow one primary code and optional secondary codes. Most failures have one dominant cause and a contributing one; forcing a single code loses information, and allowing five means nothing gets counted.

Iterate the codebook, then freeze it

The first pass through twenty cases will produce three codes you didn’t have and two that turn out to be the same thing. That’s expected — the codebook is an output of the exercise as much as an input.

Do a calibration pass: two people code the same fifteen cases independently and compare. Disagreements are either ambiguous cases or an underspecified code definition, and the second kind is fixable by writing a sharper definition with an example. The mechanics are the same as rubric calibration in writing a rubric two reviewers agree on.

Then freeze it for the round, so the counts mean one thing.

Count, weight, and rank

from collections import Counter

def rank_modes(cases, weights):
    """cases: [{'code': str, 'stratum': str}]. weights: {stratum: float}."""
    tally = Counter()
    for c in cases:
        tally[c["code"]] += weights.get(c["stratum"], 1.0)
    total = sum(tally.values())
    return [(code, n / total) for code, n in tally.most_common()]

The weights matter. If you sampled a quarter of your cases from negative-feedback traffic that is 2% of real traffic, that stratum is heavily over-represented and unweighted counts will overstate its modes. Weight by the inverse of the over-sampling, or at minimum report counts per stratum separately rather than pooled.

What comes out is a short ranked list, and in practice a couple of codes usually dominate. That concentration is the useful property: it means there’s a fix with leverage, and it’s the argument for doing this before another round of parameter tuning.

From codes to fixes

Each code maps to a different kind of work, and the mapping is the reason the taxonomy is worth the day:

  • corpus-gap → not an engineering fix. Either acquire the content or improve refusal behaviour (measuring whether your system knows when to refuse).
  • stale-doc, version-confusion → ingestion and metadata work, plus recency handling in ranking.
  • extraction-loss, boundary-split, context-orphan → the parsing and chunking layer, and a chunking change means relabelling (a re-chunk invalidates your eval set).
  • vocab-mismatch → retrieval strategy; the slice to watch when you evaluate a different embedder (testing a new embedding model).
  • duplicate-crowding, filter-overreach, rank-too-low → retrieval configuration and post-processing.
  • truncated → prompt assembly, and a snapshot test so it can’t recur silently.
  • instruction-ignored, unsupported-claim, partial-answer, over-refusal → prompt and generation work, measured end-to-end because retrieval metrics can’t see any of it (component or end-to-end).
  • ambiguous-question → product behaviour, not a bug. Clarifying questions are a feature.

Turn every coded case into a permanent test

The last step, and the one that compounds. Each coded failure becomes an eval case with its code attached: a retrieval label if the stage is upstream of generation, a must_not_say entry if the stage is generation, a must_not_retrieve entry for crowding and version cases.

Two consequences. Your eval set grows in the direction of your real failure distribution rather than your imagination. And the codes become a slice dimension, so future runs report metrics per failure mode — which lets you say “boundary-split cases went from 14 to 3” instead of “faithfulness went up a bit”. That sentence is what error analysis buys, and it’s worth more than the fifty answers took to read.

Re-run the exercise quarterly, or after any large change. The distribution moves as you fix things, and the point of a taxonomy is to keep telling you what’s dominant now rather than what was dominant when you built it.

debuggingevaluation