RAG Eval

The Five Places a RAG Pipeline Breaks

Someone hands you a bad answer. The instinct is to change the prompt, because the prompt is the easiest thing to change. Usually the prompt isn’t the problem.

A RAG pipeline has five stages that can fail, they fail differently, and there’s a fast procedure for finding which one did.

The five stages

  1. Ingestion and parsing — documents into text.
  2. Chunking — text into retrievable units.
  3. Embedding and indexing — units into vectors.
  4. Retrieval and reranking — query into a set of units.
  5. Prompt assembly and generation — units into an answer.

Failures cascade downstream and are invisible upstream, which is why diagnosis has to run backwards.

The bisection

Take the failing question. Ten minutes, in order.

Step 1: Is the answer in the corpus at all?

Search the source documents — not the index, the originals — with plain keyword search. grep, the CMS search, anything.

Not there → not a pipeline failure. The system was asked something it can’t answer, and the real bug is that it answered anyway rather than abstaining. Different fix entirely, and it’s a prompt-and-threshold problem.

There → continue.

Step 2: Is it in the index, intact?

Find the chunk that should contain the answer and print it exactly as stored.

for c in index.get_chunks(doc_id="policy-returns-v3"):
    print(repr(c.text[:400]), "\n---")

This is where a surprising share of investigations end. What you’re looking for:

  • Text mangled by parsing. Scanned PDF with no text layer, tables flattened into unaligned numbers, ligatures turned to mojibake, two-column layouts interleaved line by line.
  • The answer split across a chunk boundary. The condition is at the end of chunk 7 and the exception is at the start of chunk 8, and neither chunk answers the question.
  • Context stripped by chunking. The chunk says “30 days” but the heading that said which policy it applies to is in the previous chunk.
  • A stale version. Three copies of the policy in the index, two superseded, no version marker.
  • Boilerplate dominance. A 512-token chunk that’s 400 tokens of navigation menu and 112 of content.

Mangled or missing → stage 1 or 2. Fix the parser or the chunking strategy. No amount of retrieval tuning recovers information that isn’t in the index.

Intact → continue.

Step 3: Does retrieval find it?

Run the query directly against the retriever, at a large k, and print the ranks.

hits = retriever.search(question, k=50)
for i, h in enumerate(hits):
    print(i, round(h.score, 3), h.id, h.text[:80])

Three possible readings:

Not in the top 50. A genuine retrieval failure. Usually vocabulary mismatch — the user said “damaged,” the document says “defective” — or a query too short to embed meaningfully, or a follow-up turn that isn’t self-contained. Fixes: hybrid search (add a keyword/BM25 component alongside the dense one, which handles exact-term and rare-token cases the embedding misses), query rewriting, a different embedding model.

In the top 50 but not the top k you pass to the model. A ranking problem, not a retrieval problem, and a much better position to be in. This is what reranking exists for: retrieve broadly, rerank precisely, pass the top few. Whether it helps your corpus is measurable — retrieve at 50, rerank, and check whether precision at your generator’s k improves on your eval set.

In the top k. Retrieval did its job. Continue.

Step 4: Did the chunk reach the model intact?

Log the exact prompt string sent to the model. Not the template — the assembled text.

Things that go wrong here and are invisible everywhere else:

  • Truncation. Context assembly hit a token limit and cut the last chunk mid-sentence — often the highest-ranked one, if you assembled in reverse.
  • Deduplication removed it. A near-duplicate filter dropped the good chunk and kept its inferior twin.
  • The metadata filter excluded it. A date or permission filter that quietly matched.
  • Position. It’s chunk 18 of 20, deep in a long context, where material is used less reliably.
  • Formatting. Chunks concatenated with no separators, so the model can’t tell where one document ends and the next begins — and then attributes a claim to the wrong source.

Not in the prompt, or damaged → assembly bug. Common, easy to fix, and almost never suspected, because everyone assumes the retriever’s output equals the model’s input.

Present and intact → the last stage.

Step 5: Generation

The model had the right information and produced a bad answer anyway. Check, in order:

  • Does the prompt permit ignoring the context? “Answer using the context” is an invitation to blend. “Answer using only the context; if it does not contain the answer, say so” is a different instruction and often the whole fix.
  • Is the context contradictory? Two retrieved chunks disagreeing — typically old and new policy versions — and the model picked one, arbitrarily.
  • Is it too long? Cutting from 20 chunks to 5 good ones improves answers more often than people expect.
  • Is it a genuine reasoning failure? The answer required combining two facts and the model combined them wrongly. Real, and the rarest of the five stages in practice.

What the distribution usually looks like

Across systems, most failures land in stages 1–2 (ingestion and chunking) and stage 4 (retrieval/ranking). Stage 5 gets nearly all the attention and prompt iteration because it’s the visible surface and the cheapest thing to change.

That inversion is the single most useful thing to know about debugging RAG. When someone says “the model is hallucinating,” the prior should be that the model received garbage, not that it invented something.

Instrument it once

Doing this by hand per incident is fine for the first few. After that, trace every request:

  • the raw query and the rewritten query
  • retrieved IDs with scores, pre- and post-rerank
  • the final assembled prompt, in full
  • the raw completion
  • token counts at each stage

With that logged, step 1 through 5 becomes reading one trace instead of ten minutes of re-running. The prompt string in full is the field people omit to save space, and it’s the one that resolves the most incidents.

Then feed every diagnosed failure back into the eval set as a permanent case — building a retrieval eval set. A year of that is how a suite becomes genuinely predictive, because it accumulates precisely the cases your system got wrong.

debuggingevaluation