Which Evals Run on Every Commit
Evaluation that costs money and minutes cannot run on every commit, and evaluation that only runs before a release doesn’t catch anything while there’s still context to fix it. The resolution is to split the suite by cost and run each part at the cadence it can sustain.
The organising principle: the fast tier must be deterministic and free, and the slow tiers must be sampled and cached.
Tier 0 — invariants, on every commit
No model calls at all. Chunking produced output for every document, no chunk exceeds the embedder’s limit, metadata fields the prompt expects are present, the assembled prompt for a fixed question matches its recorded snapshot, a filtered query returns only matching documents.
Milliseconds to seconds. These are ordinary unit tests and they catch the class of bug that quality metrics notice last — see regression testing a RAG pipeline.
Tier 1 — retrieval metrics, on every commit
Retrieval scoring against labels, on a frozen fixture index. Hit rate, recall@k, precision@k, nDCG@k if you have graded labels, reported per slice.
Two properties make this the workhorse tier: it needs no LLM, so it’s free and fast, and against a frozen index it’s deterministic, so a change in the number means a change in the system. Target under two minutes for the whole thing, including embedding the eval queries — and cache those embeddings by query hash so you’re not paying for them on every run.
Everything a retrieval-only tier cannot see is real, and it’s the reason the higher tiers exist: component or end-to-end.
Tier 2 — generation and judge, nightly on a sample
Full pipeline plus judge-based faithfulness, relevance, and abstention scoring. One generation call and several judge calls per question, so a 400-question set is thousands of calls.
Three things keep it affordable:
Sample, stratified. A fixed 100-question stratified subsample, drawn once with a fixed seed and reused, gives a stable nightly trend at a quarter of the cost. Re-drawing the sample each night adds sampling noise to every comparison, which defeats the purpose — a stable subsample is a feature.
Cache aggressively. Key generations on a hash of (prompt_version, model_version, params, retrieved_chunk_ids) and judge verdicts on (judge_version, judge_prompt_version, claim, context). A commit that touches only chunking leaves most generation keys unchanged, so the nightly run costs a fraction of a cold run. Caching also removes model non-determinism from unchanged cases, which tightens every comparison.
def cached(store, key_parts, compute):
key = hash_of(key_parts) # stable hash over the full tuple
if key not in store:
store[key] = compute()
return store[key]Judge cheaply, escalate rarely. A small fast model handles the bulk of claim verification; route only the cases it flags, or where it hedges, to a stronger judge. Validate both — an escalation ladder has two instruments and two error profiles, and both belong in the report: validating an LLM judge before you trust it.
Tier 3 — the full set and human review, per release
The complete eval set through tiers 0–2, plus the human review slice, plus a look at the holdout you otherwise leave alone.
This is the tier that produces the numbers you quote to other people, and the only one where the human calibration gets refreshed. Weekly or per release; anything more frequent burns reviewer attention on noise, and reviewer attention is your scarcest measurement resource — writing a rubric two reviewers agree on.
Tier 4 — production, continuously
Unlabelled signals from live traffic, plus judge scoring on a small sampled stream, plus a canary query set run on a schedule against the real deployment.
The tier that catches what a frozen eval set structurally cannot: corpus drift, query distribution shift, silent model updates, config differences between the eval harness and the serving path. Details in monitoring RAG quality in production.
The cadence table
| Tier | What | Trigger | Model calls | Wall clock |
|---|---|---|---|---|
| 0 | invariants, prompt snapshot | every commit | none | seconds |
| 1 | retrieval metrics, all slices | every commit | none (cached embeddings) | under 2 min |
| 2 | generation + judge, fixed subsample | nightly | sampled, cached | minutes |
| 3 | full set + human slice + holdout | per release | full | hours, some human |
| 4 | production signals + canaries | continuous | sampled | ongoing |
Which tier gates a merge
Tiers 0 and 1 block. They’re deterministic and cheap enough that a failure is a real signal and a re-run is free.
Tier 2 does not block, for a practical reason: a non-deterministic check in a required status is a flaky test, and flaky required tests get bypassed. Report it as a nightly diff with an owner, and escalate when it moves beyond tolerance.
The exception worth making: if a change touches the prompt, the model, or generation parameters, tier 1 tells you nothing at all — retrieval didn’t move. Run tier 2 on the subsample for that pull request specifically. A path-based trigger in CI is enough, and it means prompt changes are never merged on the strength of retrieval metrics that couldn’t have moved.
Route by what changed
Matching the tier to the change is most of the efficiency:
- Chunking or ingestion → tiers 0 and 1, then relabel if IDs moved (a re-chunk invalidates your eval set), then tier 2 because chunk content reaching the model changed.
- Embedding model → full re-index and a paired tier 1 comparison, thresholds re-tuned before comparing (testing a new embedding model).
- Retriever, reranker, or k → tier 1 first, then tier 2 to confirm the answers moved with the metrics.
- Prompt or generation settings → tier 2 only. Tier 1 is noise here.
- Judge or its prompt → nothing about your system changed; re-validate the judge and annotate the break in the metric history.
- Nothing (a scheduled run) → tier 4 plus a nightly tier 2. This is how drift with no deploy becomes visible.
Budget it explicitly
Write the numbers down: calls per nightly run, cost per call, cost per month. Then decide the subsample size from the budget rather than discovering the bill.
A useful sanity check is the ratio of evaluation spend to serving spend. If evaluation costs a meaningful fraction of production inference, you’re either over-sampling, under-caching, or judging holistically where claim-level checks with a small model would do. And if the answer comes out uncomfortably high, the honest response is a smaller stratified subsample run every night rather than a full set run once a month — cadence beats completeness for catching regressions while the change is still fresh.