RAG Eval

Testing That Retrieval Respects Permissions

A leak does not show up as a bad score. Recall goes up when the system retrieves more, faithfulness goes up when the answer is grounded in what it retrieved, and an answer built from a document the asker was never allowed to see is, by every metric on your dashboard, a good answer.

That makes access control the one property in a RAG system that averaging cannot measure. It has to be tested as an invariant — pass or fail, per case — and the test has to know who is asking.

Your eval harness sees everything

Here is why the bug survives testing. Eval harnesses are written to be convenient: one API key, one service account, full corpus visibility, so the retrieval scores aren’t cluttered by permission filtering. Then the harness measures a system that no user has.

The minimum structural change is one field. Identity becomes part of every eval item, exactly like the query:

{
  "id": "q_211",
  "question": "what is the severance formula for a director",
  "as": "ident:contractor_eu",
  "must_retrieve": [],
  "must_not_retrieve": ["hr-comp-2026#directors", "hr-comp-2026#exec-band"],
  "expect": "refusal"
}

Note that must_retrieve is empty and must_not_retrieve is the assertion. Most permission cases are negative cases, and negative cases are the ones that aggregate recall structurally cannot express — the must_not_retrieve field from regression testing a RAG pipeline is doing the load-bearing work here.

Build the fixture around collisions

A permissions fixture that only pairs each identity with unrelated documents will pass forever without testing anything. The cases that catch real bugs are the ones where content collides:

  • The same content at two visibility levels. A public summary and a restricted full version of one policy. A filter that works and a filter that ranks-then-filters behave identically on unrelated documents and differently here.
  • Near-duplicates across tenants. Three customers on the same document template. Chunk text is almost identical, so this is where an embedding-similarity match is most likely to cross a boundary.
  • A shared document plus a restricted one that answers the question better. The correct behaviour is the worse answer, and a quality-only eval will score that as a regression. Mark these explicitly so nobody “fixes” them.
  • A document that is visible to nobody. Archived, or mid-deletion. Nothing should ever return it.
  • An identity with access to nothing. The empty-corpus case, which should produce a refusal rather than an error, an empty prompt, or a general-knowledge answer.

Three or four identities is enough. What matters is that each document in the fixture is visible to at least one and invisible to at least one.

Five layers, tested separately

A pass at the retrieval layer does not mean the answer is clean. Check each stage, because each one can leak on its own.

Retrieval. The core assertion: no returned chunk ID is outside the asker’s visible set. This one is cheap, deterministic, needs no model, and belongs in the fast tier of which evals run on every commit.

Prompt assembly. Even with clean retrieval, the assembled prompt can carry restricted text from somewhere else: a cached context, a summarised earlier turn, a static example in the system prompt, a “related documents” block built by different code. Assert against the final prompt string, not the retrieval result.

Generation. The model can restate something from conversation history that the current identity should no longer see, and it can leak metadata even when the body was withheld — a title, a filename, an author, a URL in a citation. Test the citation block as carefully as the prose; a link to a document the user cannot open still discloses that it exists and what it is called.

Refusal quality. When the answer isn’t available to this asker, the desired output is an ordinary “I don’t have that” — indistinguishable from the answer given when the corpus simply doesn’t cover it. A refusal that says you don’t have permission to see the executive compensation policy has leaked the interesting part. Score these against the abstention behaviour you already measure in measuring whether your system knows when to refuse, with the extra requirement that the two refusal types read the same.

Revocation latency. Remove an identity’s access, then re-run the probe on a schedule until it stops returning the document. The interval is a number you can put in a report. Making it shorter is your ingestion and index-maintenance problem rather than a measurement one, but nobody will work on it until somebody measures it.

Report leaks as a count

Never express this as a percentage. “99.7% of permission cases passed” invites a conversation about tolerance, and there is no tolerance: one leak is a failure, and the correct dashboard entry is an integer that is supposed to be zero.

What to report:

Field Value
Permission cases run integer, per identity
Leaks integer. Non-zero blocks the release.
Leaks by layer retrieval / assembly / generation / citation / refusal text
Revocation latency measured, per corpus source
Cases where restriction lowers answer quality expected, and enumerated

That last row is worth keeping visible for a political reason as much as a technical one. Some of your permission cases are supposed to produce a worse answer, and when a quality dashboard dips after a filter is tightened, the enumerated list is what stops somebody relaxing the filter to recover the number.

What this test does and doesn’t cover

It covers the retrieval path you tested, for the identities in your fixture, at the moment you ran it. It does not cover an identity model that is wrong in the source system — if the permission data you were handed says a contractor may read compensation documents, every layer here will pass and the system will still be doing something nobody wants.

It also cannot survive a corpus change on its own. New sources arrive with new visibility semantics, and a fixture built for three sources says nothing about the fourth. Add a case per source as it lands, and treat the identity field as permanent structure in the eval set rather than a separate suite — a leak found by the same harness that reports quality is a leak somebody actually reads, and one found in an isolated suite that runs quarterly is a leak found late. If a case does fail, reproduce it under the exact identity before touching the filter: reproducing a bad answer somebody sent you applies unchanged, with identity as the input most likely to be missing from the record.

evaluationdebugging