RAG Eval

Testing a New Embedding Model

An embedding swap is the change most likely to be evaluated badly, because it invalidates the things you were holding constant. Similarity scores land on a different scale, your refusal threshold means something else, long chunks may truncate at a different point, and the old index is unusable for comparison.

The procedure that produces a trustworthy answer: index into a parallel namespace, re-tune per-model parameters before comparing, then compare paired on the same questions.

Build a parallel index, keep the old one

Index the same corpus, with the same chunking, into a second namespace or collection. Do not overwrite. Everything except the embedding model stays identical — same documents, same chunk boundaries, same metadata, same filters.

Keeping the old index alive is what makes a paired comparison possible. Once it’s gone, all you have is a before-number from a different day, and the two aren’t comparable — different corpus state, different eval set version, possibly different k.

Verify the two indexes contain the same chunk set before scoring anything. A silent difference in document count is the most common reason an embedding comparison produces an implausible result.

Re-tune what is model-specific first

Comparing two embedders with one shared configuration measures the configuration as much as the models. Four things are model-specific and must be set per-model before the comparison:

Distance metric. Some models are trained for cosine similarity, some for inner product, and some expect vectors to be normalised. Using the wrong one degrades retrieval in a way that looks like a bad model. Check what the model documentation specifies and confirm your store is configured to match, per index.

Input truncation. Models have different maximum input lengths. A chunking scheme tuned for one model may exceed another’s limit, in which case the tail of every long chunk is silently dropped — no error, just missing content. Assert chunk length against the new model’s limit as an invariant, the same way you would in regression testing a RAG pipeline.

Query and document prefixes. Some models are trained asymmetrically and expect a short instruction or prefix on queries that differs from the one on passages. Omitting it, or applying the document form to queries, costs real retrieval quality. This is a per-model detail to read from the model’s own documentation rather than assume; getting it wrong is a common cause of a new model “performing worse”.

Score thresholds. Any absolute similarity cutoff — a refusal threshold, a filter on weak results — is calibrated to the old model’s score distribution and is meaningless under the new one. Re-tune it against your eval set before comparing, and note that this alone can reverse the conclusion.

Also re-tune k if you use a threshold-free top-k: the two models may need different depths to reach comparable recall, and comparing at a single k chosen for the incumbent favours the incumbent.

Then compare, paired

Run both indexes over the same eval questions and compare per question.

def compare_indexes(a, b, eval_set, k, relevant):
    """relevant(chunk, item) -> bool. Returns per-question recall for both."""
    rows = []
    for item in eval_set:
        if not item["answerable"]:
            continue
        ra = recall(a.search(item["question"], k=k), item, relevant)
        rb = recall(b.search(item["question"], k=k), item, relevant)
        rows.append((item["id"], ra, rb))
    return rows

Report wins, losses and ties rather than two means. An embedding change is almost always a trade — better on paraphrase, worse on exact identifiers, or the reverse — and the mean conceals which trade you’re taking. The reasoning is in was that a real improvement or noise?.

Then read the losses. Fifteen minutes with the questions that got worse tells you more than the aggregate does, and it usually names the pattern: product codes, acronyms, negations, non-English queries, very short queries.

Slice it, especially by vocabulary

The slices that matter most for an embedding decision:

  • Vocabulary-mismatch questions, where the user’s word differs from the corpus’s. This is the main thing dense retrieval is supposed to buy, so it’s the slice where a better model should show up.
  • Exact-identifier questions — part numbers, error codes, proper nouns. Frequently where a new model regresses, and frequently the queries that matter most operationally.
  • Query length, short versus long.
  • Language, if your corpus or users are multilingual. A model’s behaviour outside its primary training languages can differ sharply from its headline behaviour, and an aggregate over a mostly-English set won’t show it.
  • Document type, since some models handle tabular and code-like text markedly better than others.

Per-slice reporting is what turns “model B is better” into “model B is better except on part numbers, where it’s worse, so we need hybrid retrieval or we lose a class of query”. More on the mechanics in the aggregate score hides the bug.

Finish end-to-end

Retrieval metrics are the screening stage, not the decision. Once a candidate wins on paired retrieval, run generation and judge scoring on a sample through both indexes: faithfulness, answer relevance, abstention behaviour.

Two reasons this step is not optional. Retrieval gains don’t automatically become answer gains — if the newly-retrieved chunk lands at rank 8 and the generator reads five, nothing changed. And the abstention profile shifts with the score distribution, so a swap can quietly make the system answer more unanswerable questions even when recall improved. Check the confusion matrix from measuring whether your system knows when to refuse on both arms.

The four ways this test goes wrong

Comparing against a stale baseline. The old number came from a different corpus state or eval set version. Always re-run the incumbent today, on the same set, in the same session.

Leaving a threshold at its old value. Guarantees a wrong conclusion. Re-tune per model, and say in the report which thresholds each arm used.

One shared distance metric or missing prefixes. Configuration error presenting as model quality.

Evaluating only the aggregate. Hides the trade, and the trade is the decision.

What this post deliberately doesn’t cover

Reindex time, storage growth from a larger vector dimension, and inference cost are real inputs to an embedding decision and they’re not measurement questions. This procedure tells you the quality difference; the economics of acting on it are a separate conversation and belong elsewhere.

One measurement note about dimension, though: a model with more dimensions is not automatically better on your corpus, and the paired comparison is how you find out rather than assuming. Run the smaller candidate too — if it ties, the decision gets easier for reasons that have nothing to do with retrieval quality.

retrievalevaluation