Component or End-to-End: Which Evaluation Answers Which Question
There are two ways to evaluate a RAG system and they answer different questions. End-to-end evaluation scores the final answer and tells you whether the system is doing its job. Component evaluation scores one stage in isolation and tells you which stage to work on.
Teams that run only one of them get stuck in a predictable way. End-to-end only: you know it’s bad and have no idea why. Component only: every number is green and users still complain.
What each one actually measures
End-to-end takes a question, runs the whole pipeline, and scores the answer. Faithfulness, answer relevance, correctness against a reference, a human rubric score — all end-to-end metrics. They are the only metrics that correspond to something a user experiences.
Component takes a question and scores one stage’s output against a label for that stage. Recall@k and precision@k score the retriever. Chunk-boundary integrity scores the splitter. Rank position of the known-relevant chunk before and after reranking scores the reranker. None of these say anything about the answer.
The relationship between them is asymmetric and worth stating precisely: component metrics bound end-to-end quality but do not determine it. If the relevant document never enters the context, no generation setting recovers it — retrieval recall is a ceiling. But retrieval recall at 1.0 licenses no conclusion at all about answers, because the model still has to read, weight, and not embellish.
That asymmetry is why the two are not interchangeable, and why “retrieval looks fine” is not a defence.
The failure of end-to-end alone
An end-to-end score is a single number covering five or six stages. When it drops, the search space for the cause is the whole pipeline, and the usual response is to guess: change the prompt, bump k, swap the embedding model, all at once, and watch whether the number moves.
This produces motion without knowledge. If the number improves you don’t know which change did it, and if two changes cancel out you conclude neither helped.
End-to-end evaluation is also slow and expensive, because every question means a generation call and usually a judge call on top. That cost pushes teams toward small sets, and small sets can’t resolve small differences — see was that a real improvement or noise?.
The failure of component alone
The opposite failure is quieter. Retrieval recall@10 improves, everyone celebrates, and the answers are identical — because the generator only reads five chunks, or because the newly-retrieved chunk landed at rank 9 where the model barely attends to it, or because the material was already covered by another retrieved chunk.
Component metrics are also easy to make look good in ways that don’t transfer. Raising k raises recall monotonically and often makes answers worse, since it dilutes the context with plausible-but-irrelevant material. Optimising a component metric without an end-to-end check is optimising a proxy.
Run both, and connect them with ablations
The connective tissue between the two levels is the ablation: hold everything constant except one stage, and measure the end-to-end effect.
The most useful single ablation is the oracle context run. Instead of retrieving, hand the generator exactly the labelled relevant chunks from your eval set, and score the answers.
def oracle_run(generate, eval_set, corpus):
"""End-to-end scoring with retrieval replaced by ground truth."""
for item in eval_set:
if not item["answerable"]:
continue
context = [corpus[doc_id] for doc_id in item["relevant_doc_ids"]]
yield item["id"], generate(item["question"], context)Two readings, both decisive:
- Oracle answers are good, live answers are bad → the problem is upstream of generation. Work on retrieval, chunking, or the index.
- Oracle answers are also bad → retrieval is not your bottleneck. The prompt, the model, or the chunk content is. Fixing retrieval will not help, and this is the check that saves weeks of it.
Other ablations worth having ready: retrieval with reranking disabled, generation with the context shuffled (tests position sensitivity), and generation with one irrelevant chunk deliberately injected (tests distractor robustness).
A reporting layout that keeps them apart
| Level | Metrics | Cadence | Cost |
|---|---|---|---|
| Retrieval (component) | hit rate@k, recall@k, precision@k, nDCG@k | every commit | seconds, no model calls |
| Generation (component, oracle context) | faithfulness, answer relevance | nightly | one generation + judge per question |
| End-to-end (live pipeline) | faithfulness, relevance, correctness, abstention | nightly and per release | same, plus retrieval |
| Human | rubric score on a sample | weekly | expensive, and the calibration for everything above |
The generation row is the one most teams don’t have, and it’s the one that separates “the model can’t use good context” from “the model never got good context”. Running it with oracle context is what makes the row interpretable.
Report every level with the k and the model version attached. A faithfulness number without the retrieval configuration it was measured under isn’t comparable to next month’s.
Which to reach for, by symptom
A stakeholder shows you one bad answer. Trace that single question through both levels — component first, since it’s cheap. The five places a RAG pipeline breaks is the procedure.
You’re choosing between two retrievers. Component metrics for the shortlist, end-to-end on a sample for the final call. Component alone will pick the one that maximises recall@50, which may not be the one that produces better answers at your production k.
You changed a prompt. End-to-end only. Retrieval didn’t move; measuring it again is noise.
You re-chunked. Both, and re-label first, because your labels reference chunk IDs that no longer exist — a re-chunk invalidates your eval set.
Quality is drifting in production with no code change. Neither, initially. Component and end-to-end evaluation both run against a frozen set that can’t see corpus or model drift; look at production signals instead — monitoring RAG quality in production.
What the numbers license you to say
With component metrics only: “the retriever surfaces at least one labelled relevant chunk in the top 5 for 84% of our eval questions.” Nothing about answers. (Illustrative figure.)
With end-to-end only: “on our eval set, 12% of answers contained a claim the retrieved context doesn’t support.” Nothing about which stage to fix. (Illustrative figure.)
With both plus an oracle ablation: “answers fail primarily because the relevant material isn’t retrieved for multi-document questions; given the right context, the generator handles them.” That sentence is a plan, and it takes all three runs to earn it.