Measuring Whether Your System Knows When to Refuse
The worst answer a RAG system can produce is a confident one to a question its corpus cannot answer. Most eval sets can’t detect it, because they contain only questions with answers — every question is answerable, so answering is always right, and the metric has no way to reward restraint.
Abstention is a two-error classification problem, and it needs its own labels and its own confusion matrix.
The two errors
| System answered | System refused | |
|---|---|---|
| Answerable | correct behaviour | over-refusal |
| Unanswerable | over-answering | correct behaviour |
Over-answering is the dangerous error. The user gets a fluent, plausible answer with no support, often with citations to documents that discuss the topic without containing the fact. Trust in the whole system is downstream of this cell.
Over-refusal is the annoying error, and it’s the one you create while fixing the first. A system tuned to refuse whenever retrieval scores are middling refuses questions it could answer, users stop asking, and the failure is invisible in complaint logs because people just leave.
You need both numbers or you’ll trade one for the other without noticing. Reporting only “hallucination rate went down” after tightening a threshold is reporting half of a trade.
Building the unanswerable set
Unanswerable questions have to be constructed, because logs mostly contain questions users expected to work. Aim for 10–20% of the eval set, spread across these kinds — they fail differently and a single kind gives you a misleading read.
Out of scope. Plainly about something else. The easy case; a system that fails these is badly broken.
In-domain but absent. The corpus covers returns policy and says nothing about international returns. Retrieval will confidently return the domestic policy, which is topically perfect and factually irrelevant. This is the hard case and the one that matters most.
False premise. “How do I claim the third-year extended warranty?” when there is no third-year warranty. The correct answer corrects the premise; the common failure invents the procedure.
Time-shifted. Questions about a policy that changed, or a future period the corpus doesn’t cover. Retrieval finds the old version and reports it as current.
Permission-shifted. Questions whose answer exists in the corpus but not in the documents this user may see. If your retrieval is permission-filtered, the correct behaviour for an unauthorised user is a refusal, and this is worth testing explicitly because it’s a security property expressed as an eval case.
Underspecified. “How much does it cost?” with no product named. The correct behaviour is a clarifying question, which is a third output class and should be labelled as such rather than as a refusal.
Record why each is unanswerable. Scores by reason are actionable; an aggregate abstention rate isn’t.
Detecting a refusal, which is harder than it sounds
The scoring trap: classifying the output as answer-or-refusal by string matching. Models rarely refuse cleanly. Far more common is the hedged non-refusal — “Based on the available information, returns are generally accepted within 30 days, though you may wish to confirm with support.” That is an answer with a disclaimer, and a keyword check for “I don’t know” or “not covered” scores it as a refusal.
Classify with three verdicts instead:
REFUSAL_CHECK = """Question: {question}
Response: {answer}
Classify the response:
REFUSED - states the available sources do not contain the answer
ANSWERED - asserts an answer, including with hedging or caveats
CLARIFY - asks the user for more information instead of answering
Reply with exactly one word."""Hedging counts as ANSWERED. This is a deliberate strictness: a user reading a hedged answer takes the answer, not the hedge, and grading the caveat as restraint lets a real failure through. Validate this classifier on a handful of hedged examples before trusting the rate — the general procedure is in validating an LLM judge before you trust it.
Also score refusal quality on the correct refusals. “I don’t have information about international returns in the available documents” is useful. A bare “I cannot answer that” is a support ticket. Only the first tells the user whether to rephrase or go elsewhere.
Reading the numbers
Four rates, from the matrix:
- Over-answering rate = answered / unanswerable questions.
- Over-refusal rate = refused / answerable questions.
- Refusal informativeness = correct refusals that name what’s missing / all correct refusals.
- Clarification rate on underspecified questions, scored separately in both directions.
The pair to watch is the first two together. Any lever you pull moves both, usually in opposite directions, and a change is only an improvement if one drops without the other rising. Plot them as a pair across settings rather than reporting either alone.
The levers, and what each does to the pair
Prompt instruction. “Answer using only the provided sources; if they do not contain the answer, say so explicitly and name what is missing” versus “answer using the provided sources”. The cheapest lever and often the largest effect. It also raises over-refusal, especially on questions whose answer requires a small inference from the sources.
A retrieval score threshold. Refuse when the top result’s score is below a cutoff. Intuitive and blunter than it looks: similarity scores are not calibrated probabilities, their scale is specific to the embedding model, and the threshold that worked before an embedding swap is meaningless after one — see testing a new embedding model. Tune it against the matrix, on your data, and re-tune it whenever the model or corpus changes.
A sufficiency check before generating. Ask a model whether the retrieved context is sufficient, and refuse if not. More accurate than a score threshold and it costs an extra call per query. Note that it’s the same check described in evaluating retrieval before you have labels, used at serving time instead of eval time.
Post-hoc claim verification. Generate, verify each claim against the context, and suppress or flag the answer if support is weak. The most accurate and the most expensive, and it converts some over-answering into visible uncertainty rather than into refusal.
None of these eliminate unsupported answers. Each shifts the error distribution: threshold-based refusal trades unsupported answers for refused answerable questions; verification trades them for latency and cost. Say which trade you made when you report the improvement.
Where abstention sits in production
Offline abstention rates come from a set you built. In production the mix of unanswerable questions is different and drifting, and you have no labels — so track the refusal rate as a monitored time series instead. A step change in it, with no deploy, means the corpus or the query distribution moved. That’s a leading indicator worth alerting on: monitoring RAG quality in production.