About RAG Eval
RAG Eval is an independent site about evaluating and debugging retrieval-augmented generation systems.
Not a product, not a vendor, no affiliate links, nothing to buy. No tool rankings.
Why evaluation, specifically
Because building a RAG prototype is now easy and knowing whether it’s any good is still hard, and the gap between those two facts is where most projects stall.
The prototype takes an afternoon: embed some documents, store the vectors, retrieve the top few, put them in a prompt. It demos well, because you tested it with questions you wrote while looking at the documents. Then it meets real users, who ask things obliquely, use vocabulary the corpus doesn’t, expect answers that span four documents, and ask questions the corpus simply cannot answer — and the system, which has no notion of not knowing, answers anyway.
At that point you need to distinguish “retrieval didn’t find it,” “retrieval found it and the model ignored it,” “the document was parsed into nonsense six weeks ago,” and “the answer isn’t in the corpus at all.” Those are four different problems with four different fixes, and no amount of prompt tweaking will tell you which one you have.
That’s what this site is for.
What we think
Retrieval, generation, and end-to-end are three separate measurements. Retrieval metrics ask whether the right context was found. Generation metrics ask what the model did with the context it got. End-to-end metrics ask whether the user got a good answer. A system can score well on any one and badly on the others, and averaging across the boundary produces a number that moves for reasons you can’t trace. Keeping them apart is the single most useful discipline in this field.
Most eval sets are rigged, unintentionally. If you wrote the questions with the source document open, you wrote questions whose phrasing matches the document’s phrasing, which is the easiest possible retrieval case. Real users don’t have the document open. Eval sets built this way produce reassuring numbers that don’t survive contact with anyone.
Faithfulness is not quality. An answer can be fluent, useful, and entirely unsupported by the retrieved context — the model filled a gap from its parameters. Sometimes the result is even correct, which makes it harder to catch. If you’re not measuring groundedness separately, you’re not measuring the thing that made you choose retrieval in the first place.
LLM-as-judge is necessary and imperfect. For open-ended answers it’s the only thing that scales. It also exhibits documented biases — toward longer answers, toward its own outputs, toward whichever option is presented first. Use it, calibrate it against human labels on a sample, and never quote its scores as though they were measurements.
Retrieval doesn’t eliminate hallucination. It changes the error distribution: fewer invented facts about things in your corpus, and a new class of errors where the model confidently synthesises across retrieved chunks that shouldn’t have been combined. Anyone selling RAG as a hallucination fix is describing half the change.
On numbers
We don’t publish benchmark figures as though they generalised. Whether reranking helps your system, how much chunk size matters, what recall@k you should expect — these are properties of your corpus and your query distribution, not constants. The honest deliverable is the mechanism plus a procedure for measuring it on your own data.
Worked examples appear throughout with hypothetical numbers, always labelled. We don’t invent paper titles, published scores, or library methods; where a technique has a well-known name we use it and describe the idea, rather than manufacturing a citation.
Elsewhere in this network
Learning what RAG is — what-is-rag.com (beginner Q&A) and retrieval-augmentedgeneration.com (plain-language guide). Precise term definitions — whatisrag.net. Whether retrieval is the right architecture at all, and what it costs — what-is-rag.org. This site assumes you’ve already built the thing.