RAG Eval

A Re-Chunk Invalidates Your Eval Set

Change your chunking and every relevance label referencing a chunk ID becomes a pointer to something that no longer exists. The eval run still completes, the numbers still print, and they describe nothing — a retriever scored against dead IDs looks catastrophically worse, and a retriever scored against partially-surviving IDs looks arbitrarily different.

This is the most common way a RAG eval set silently stops measuring, and the fix is to stop labelling chunks.

Why chunk IDs are the wrong anchor

A chunk is an artefact of a splitting decision, not a fact about your corpus. Change the size, the overlap, the separator rules, or the parser, and every boundary moves. Even a deterministic scheme produces different chunks after a document is edited or the extraction library is upgraded.

Labels anchored to those artefacts inherit their instability. Worse, the failure is partial: some IDs survive, some don’t, and a partial failure produces a plausible number instead of an error. A hard crash would be kinder.

Label spans in documents

Anchor labels to the document and the character range containing the answer.

{
  "id": "q_014",
  "question": "how long do I have to return a damaged item",
  "relevant_spans": [
    {"doc": "policy-returns", "doc_version": "v3", "start": 4120, "end": 4388},
    {"doc": "faq-shipping",   "doc_version": "v2", "start": 902,  "end": 1105}
  ]
}

A span survives re-chunking because it describes the corpus, not the index. Scoring then becomes a containment test: a retrieved chunk counts as relevant if it overlaps a labelled span by enough to carry the answer.

def chunk_is_relevant(chunk, spans, min_overlap=0.5):
    """chunk: (doc, start, end). Relevant if it covers enough of any span."""
    doc, c0, c1 = chunk
    for s in spans:
        if s["doc"] != doc:
            continue
        covered = max(0, min(c1, s["end"]) - max(c0, s["start"]))
        if covered / (s["end"] - s["start"]) >= min_overlap:
            return True
    return False

The min_overlap threshold is a real modelling choice, not a detail. A chunk containing 40% of the labelled span may or may not contain the answer, and the two error directions differ: too low and you credit chunks that don’t answer the question, too high and you penalise a legitimately split answer. Pick a value, record it beside the metric, and never compare numbers computed with different values. Half of an answer span is a reasonable default for prose; for tabular content, require the whole span, because half a table row is not an answer.

This also gives you a free diagnostic. If a chunking change leaves many chunks overlapping a span partially and none covering it, your splitter is cutting answers in half — a specific, fixable finding that chunk-ID labels cannot express.

Recovering labels you already have

If your labels are chunk IDs and you kept the old chunk text, you can migrate rather than relabel.

1. Reconstruct spans from old chunks. Locate each old chunk’s text in its source document to get character offsets. Exact string search works for most; normalise whitespace first, and expect a tail of failures where extraction changed between runs.

2. Map spans onto new chunks with the overlap test above.

3. Split the results into three buckets. Cleanly mapped, ambiguously mapped (several new chunks partially overlap, none dominantly), and unmapped.

4. Trust the first bucket, review the other two by hand. The ambiguous ones are where the interesting information is: they mark exactly the places the new chunking fragments an answer.

Budget for the manual portion. Migration is cheaper than relabelling from scratch and it is not free, which is itself an argument for span labels — you pay this cost once.

The trend line breaks, and should say so

Even with a perfect migration, metrics computed before and after a chunking change are not strictly comparable: the retrieval units changed, so recall@5 now means “five different-sized things”.

Handle it explicitly rather than pretending continuity:

  • Annotate the break in the metric history with the corpus and chunking version. A trend chart with an unmarked discontinuity is worse than two charts.
  • Run both configurations once on the same eval set, at the same time, and record both numbers. That paired run is the only honest bridge between the two eras, and it takes one extra evaluation to get — details in was that a real improvement or noise?.
  • Keep the old index around until the paired run is done. Deleting it first means the comparison can never be made.

Version the set with the corpus

An eval set is a claim about a specific corpus state. Store the two together:

  • corpus snapshot ID and document versions
  • chunking configuration, including the parser and its version
  • embedding model and version
  • the label schema version and the min_overlap used

Then a stale run is detectable instead of merely wrong: if the recorded corpus version doesn’t match the index you’re evaluating, the harness should refuse to run. That check is worth adding today — it converts the whole class of silent-mismatch failures into a loud one, which is the standing rule in regression testing a RAG pipeline.

Document edits need the same treatment at a finer grain. When a document changes, its spans may shift by hundreds of characters, so store a short text anchor — the first and last few words of the span — alongside the offsets and re-locate on mismatch. Offsets alone rot on every edit; offsets plus an anchor usually survive.

What this doesn’t fix

Reference answers still go stale. If the policy itself changed, the correct answer changed, and no span mechanism detects that. Version reference answers against document versions and re-verify on update — scoring answers against a reference.

Deleted documents. A span pointing into a document that no longer exists is unanswerable now, which may be correct behaviour rather than a regression. Retire those cases deliberately, or convert them into abstention cases, which is often the more valuable outcome — measuring whether your system knows when to refuse.

New relevant material. Labels list what was relevant when you labelled. Documents added since may also contain the answer, and a retriever surfacing them gets marked wrong. This is unavoidable and it argues for periodic re-labelling passes over the questions you care most about, and for reading the false positives rather than trusting the count — the labelling discipline in building a retrieval eval set applies every time the corpus grows substantially.

evaluationretrieval