Answer Relevance Is Not Answer Correctness
Answer relevance asks one narrow thing: does the response address the question that was asked? Not whether it’s true, not whether the context supports it — whether it’s an answer to this question. It fails independently of the other two, and a single blended “quality” score hides that.
The characteristic failure is an answer that is entirely accurate, entirely grounded in your documents, and about a neighbouring topic. Users experience it as the system not listening.
Where relevance sits among the metrics
Four properties, four different measurements:
Faithfulness — every claim is supported by the retrieved context. A relationship between the answer and the context. Covered in measuring faithfulness, not just answer quality.
Correctness — the answer is true. Requires a reference or an expert.
Relevance — the answer addresses the question. A relationship between the answer and the question, and the only one of the four that never looks at the corpus.
Quality — length, structure, tone, hedging. Presentation.
Relevance is the cheapest to measure and the one most often left out, because faithfulness pipelines compare the answer to the context and never re-read the question.
The four ways relevance fails
The adjacent answer. Asked about refunds for damaged goods, the system explains the standard 30-day return window. Retrieval found the returns policy, which is topically close and doesn’t cover the case. Faithful, true, not an answer. This is the most common relevance failure and it correlates strongly with vocabulary mismatch between the question and the corpus.
The partial answer. A two-part question gets one part answered. “Can I expense this, and who approves it?” produces a confident paragraph on expensing and silence on approval. Scored holistically it reads as fine, because the part that’s there is good.
The padded answer. The answer is in there, wrapped in three paragraphs of adjacent policy that the model included because it was in the context. Relevance scoring should penalise this — not for style reasons, but because the user has to find the answer inside it, and because padding is a reliable symptom of a low-precision retrieval set.
The evasive answer. “This depends on several factors and you should check with your administrator” when the corpus does contain the answer. An abstention where none was warranted. Measured properly this belongs with abstention, but it shows up in a relevance score as a low value with no faithfulness violation, which is a useful signature.
Measuring it
The tractable procedure mirrors claim decomposition, but decomposes the question rather than the answer.
1. Split the question into the things it asks for. Most questions have one. Some have two or three, and those are the ones that fail.
2. For each sub-question, decide whether the answer addresses it. Three verdicts: answered, partially answered, not addressed.
3. Score. Relevance = addressed / total sub-questions, with partial counting as a half. Track “questions with any unaddressed part” separately — it’s the number that matters for multi-part questions, and averaging buries it.
DECOMPOSE_Q = """List the distinct things this question asks for.
One per line. If it asks for exactly one thing, output one line.
No commentary.
Question: {question}"""
ADDRESSED = """Question part: {part}
Answer: {answer}
Does the answer address this part of the question?
Reply with exactly one word: ADDRESSED, PARTIAL, or MISSING.
Judge only whether it is addressed. Do not judge whether it is correct."""The last instruction is load-bearing. Without it a judge conflates relevance with correctness and you lose the separation you built the metric for. Test it: hand it a confidently wrong answer that squarely addresses the question, and confirm it says ADDRESSED.
The reverse-question trick, and its limits
A widely-used alternative: ask a model to generate the question that the answer appears to be answering, then compare that generated question to the real one by embedding similarity. High similarity implies the answer was on-topic.
It’s cheap and it catches gross topic drift. Two reasons not to rely on it alone:
- Embedding similarity is topical, not logical. “How long do I have to return a damaged item?” and “How long do I have to return an item?” embed extremely close together, and the difference between them is the entire failure case.
- It rewards restating the question. An answer that opens by paraphrasing the question generates a near-identical reverse question and scores well regardless of what follows.
Use it as a cheap screen in a fast eval tier, with sub-question checking as the metric you actually report. The tiering argument is in which evals run on every commit.
Reading relevance together with the others
The pairing is where the diagnostic value is.
| Faithfulness | Relevance | Likely cause |
|---|---|---|
| high | high | working |
| high | low | retrieval found adjacent material; the model faithfully summarised the wrong thing |
| low | high | the model answered from parametric knowledge, ignoring the context |
| low | low | context was useless and the model improvised |
The high-faithfulness/low-relevance cell is the one this metric exists for. A faithfulness-only pipeline scores those answers as successes, and they are the failures users complain about most, because the system sounds authoritative while missing the point.
Slice relevance by question type as well as reporting the mean. Multi-part questions will score visibly worse than single-part ones, and that gap is actionable — it usually points at a retrieval set assembled for the first clause only. More on slicing in the aggregate score hides the bug.
What a low relevance score does not tell you
It doesn’t tell you the stage. An adjacent answer can come from retrieval (the right chunk wasn’t found), from chunking (the chunk containing the specific case got split from its heading), or from generation (the material was there and the model summarised the general rule instead).
Distinguish with the oracle-context run: hand the generator the labelled relevant chunks and re-score relevance. If it recovers, the problem is upstream. If it doesn’t, the prompt is inviting generality — instructions like “answer the question directly and completely; if part of the question is not covered by the context, say which part” measurably change this behaviour, though how much depends on the model and you should verify it on your own set rather than assume it.