The Chunks Your Retriever Never Returns

Your recall@10 is healthy, your judge scores are stable, and a subject-matter expert keeps saying the system “doesn’t know about” a part of the corpus that you can confirm is indexed. Both are true at once, and no metric on your dashboard is going to reconcile them.

Eval sets measure the questions you thought to ask. A coverage audit measures the corpus from the other end: what fraction of what you indexed has ever come back from a search, and of the rest, how much could.

Retrieval frequency is extremely skewed

Take a month of production queries and count the distinct chunk IDs that appeared in any top-k. Divide by the number of chunks in the index. Whatever that ratio is on your system, it will be lower than you expect, and most of the gap is legitimate: users ask about a fraction of what you store.

The interesting part is the tail’s composition. A chunk that has never been returned is in one of two states, and telling them apart is the entire exercise:

• Never asked about. Nobody has posed a question it answers. Fine, and not actionable.
• Not retrievable. No query can surface it, including a query built from its own text. That is a defect, and it is silent — the document is in the index, the ingestion report says success, and the aggregate metrics are computed over questions whose answers live elsewhere.

Probe the never-returned set

The separator is a known-item probe, the same instrument as in evaluating retrieval before you have labels: build a query out of the chunk’s own most distinctive text and check whether the chunk itself comes back.

def reachable(chunk, retriever, k=10):
    probe = distinctive_span(chunk.text)      # a rare phrase, a title, an ID
    hits = retriever.search(probe, k=k)
    return chunk.id in {h.id for h in hits}

If a chunk cannot be retrieved by a near-verbatim quotation of itself, no user query will ever find it. Run this over a sample of the never-returned set rather than all of it — a few hundred chunks is enough to estimate the unreachable share and to surface the pattern, which is what you actually want.

Group the failures by anything you recorded at ingestion time: source system, file type, ingestion date, language, tenant, pipeline version. Unreachability almost never scatters randomly. It arrives in clumps, and the clump names the cause.

What the clumps usually turn out to be

Six causes, and the check for each:

The vector is degenerate. An embedding call that failed, was retried into a placeholder, or was computed on an empty string leaves a vector that is either zero or identical across many chunks. Check for duplicate and zero vectors directly; also look for chunks that appear in an implausibly large share of result sets, because a degenerate vector is often equidistant from everything and shows up as a hub.

The text was truncated before embedding. The model has an input limit, and content past it was silently dropped. Symptom: chunks are reachable by a phrase from their first paragraph and unreachable by anything from their last.

The chunk has no distinctive words. Navigation, headers, a page of table borders, a caption with no context. There is nothing to match. The remedy is a rule in your chunking or extraction stage, which is not this site’s subject — the measurement contribution here is that a coverage audit produces the list, ranked by how much of the index it accounts for.

A filter excludes it. The most common version: your query path always filters on a metadata field, and a batch of documents was ingested without that field. They are in the index and structurally invisible. Reproduce by running the same probe with filters off; if it is reachable unfiltered, you have found it.

Near-duplicate cannibalisation. Five near-identical copies of a policy compete for the same query. One wins consistently and the others are never returned — which is harmless for answering and actively misleading for a coverage audit, so resolve duplicates into equivalence groups before you count anything.

Nobody who queries can see it. A tenant or permission scope with content and no traffic. Real, and not a retrieval bug.

Report it as a defect count, not a score

Coverage is not a quality metric and should never be put on the same chart as recall. It is a defect finder, and the useful reporting shape is:

Number	Meaning
Share of index returned at least once in 30 days	Context only. Not a target.
Unreachable share of the never-returned sample	The defect estimate. Should trend to zero.
Unreachable chunks by cause	The work queue.
Documents fully unreachable (no chunk reachable)	The worst cases. Fix first.
Hub chunks: appearing in >x% of result sets	Suspected degenerate vectors.

The fourth row is the one to escalate. A document with one unreachable chunk out of forty is a nuisance; a document where nothing is reachable is content you believe you have and do not.

Do not turn the first row into a target. Retrieval frequency is driven by what users ask, and optimising it means promoting chunks nobody wants. That way lies a retriever tuned to spread its results around, which is worse at its job by every measure that counts.

Turning the audit into a standing check

Once, as an audit, it produces a work queue. Wired into ingestion, it prevents the recurrence: after each batch lands, probe a sample of the new chunks with their own text and fail the run if the unreachable share exceeds a small tolerance. This is the cheapest possible check — no labels, no judge, no LLM at all — which makes it a natural fit for the invariant tier in which evals run on every commit.

It also closes a gap that neither of your other instruments can. An eval set can only fail on questions it contains, and production signals can only fail on traffic that arrives. A chunk that is unreachable and unasked-about is invisible to both, right up until the day somebody finally asks — and then it presents as the confident wrong answer, or the refusal, that you will spend an afternoon bisecting through the five places a RAG pipeline breaks.

Worth adding once you have the numbers: coverage by ingestion date, on the same chart as your quality trend. A step change in unreachable share that lines up with a pipeline deploy is the clearest causal evidence this audit produces, and it is the reason to keep the history rather than just the current figure.