Thumbs Down Is a Weak Label
Thumbs up and thumbs down are the cheapest quality signal available and the most frequently over-read. The response rate is tiny, the people who respond are not a random sample, and the failure that matters most — a confident, fluent, wrong answer — reliably collects thumbs up.
Feedback is genuinely valuable. It’s valuable as a sampler that points you at cases worth reading, not as a metric you trend and report.
Five reasons the number doesn’t mean what it says
Sparse and self-selected. Only a small share of users ever click, and the ones who do are systematically different from the ones who don’t — more engaged, more annoyed, or more diligent. Whatever rate you observe describes them.
Skewed to extremes. People rate when something is unusually good or unusually bad. The mediocre middle, which is where most quality problems live, is nearly unrepresented.
Ambiguous target. A thumbs down can mean the answer was wrong, incomplete, slow, badly formatted, about the wrong product, or that the underlying policy is unpopular. A single bit cannot distinguish “you’re broken” from “I don’t like the answer you correctly gave me”.
Blind to the important failure. A fluent, well-cited, plausible answer that is wrong gets thumbs up from a user who couldn’t have known. Feedback measures perceived quality, and the gap between perceived and actual quality is exactly the thing RAG evaluation exists to close.
Response propensity moves. Change the widget’s position, its wording, or the answer’s length, and the click rate changes without any change in quality. This makes feedback rate a poor time series: a UI experiment can produce what looks like a quality regression.
Instrument it so a click carries information
You can make the signal substantially better for very little product work.
Ask for a reason, from a fixed list. Four or five options that map to your failure taxonomy: wrong, incomplete, not what I asked, sources don’t support it, outdated. Coarse, fast to click, and it turns one bit into a routable category — reasons that align with the codes in a failure taxonomy for RAG answers are directly countable.
Offer an optional free-text box. The minority who fill it in produce the highest-value evaluation input you will ever get, because they explain the failure in the user’s own terms.
Capture the trace with the click. Query, session context, retrieved chunk IDs and scores, prompt version, model version, arm. A thumbs down without the trace is an anecdote; with the trace it’s a debuggable case. Reconstructing it later is often impossible because the corpus and index have moved on.
Log positive feedback with the same detail. Not for the ratio — for the contrast. Cases users liked that your judge scored badly are where your metric is wrong, and that’s more valuable to find than one more disliked answer.
Use it as a sampler
The right place for feedback in an evaluation programme:
Triage queue. Negative feedback, ordered by frequency of the underlying query, is your highest-yield reading list. Frequency matters — one user’s complaint about a rare query is worth less than a common query quietly failing for everyone.
Stratum in error analysis. Negative-feedback cases are one stratum among several, weighted down because they’re heavily over-sampled relative to real traffic. Pooling them unweighted with random samples overstates their failure modes.
Source of eval cases. Every confirmed failure becomes a permanent case, with the class of failure captured rather than just the exact query.
Metric-disagreement detector. Cases where feedback and your automated metrics conflict, in either direction, are the ones that improve your metrics. High faithfulness with a thumbs down usually means the answer was faithful and irrelevant — see answer relevance is not answer correctness.
Pair it with implicit signals
Behavioural signals are lower-quality per event and vastly more plentiful, and the combination is stronger than either.
- Reformulation immediately after an answer — the best cheap negative signal.
- Escalation to a human or to searching the documents directly.
- Copying the answer, or opening a cited source, both weak positives with the caveat that citation clicks can mean distrust.
- Dwell before the next action, which is noisy and occasionally informative at scale.
The details of what each proxy confounds with are in A/B testing a retrieval change. The reason to pair them with feedback: implicit signals cover all traffic, so they’re not self-selected, and explicit feedback tells you what the implicit signal meant on the small number of cases where you have both.
Track feedback rate as its own metric
Not as a quality metric — as a diagnostic on the instrument.
If the share of answers receiving any feedback changes, something about the UI, the answer format, or the user population changed, and every ratio computed from feedback moved for reasons unrelated to quality. Monitor the denominator alongside the ratio, and annotate UI changes on the chart. This is the same discipline as recording judge versions with judge scores: an instrument’s own behaviour is part of the measurement (validating an LLM judge before you trust it).
What to report, and what not to
Don’t report “94% positive feedback” as a quality figure. On a small self-selected sample it’s uninformative, and it will be quoted back at you the week a real regression lands with the number unchanged.
Do report the count of distinct issues found via feedback, the categories they fell into, how many became eval cases, and — separately — the sampled judge score and the human review score, which are your actual quality numbers (monitoring RAG quality in production).
The honest framing for a stakeholder: feedback tells us what to look at, sampled judging tells us how we’re doing, human review tells us whether the judging is right. Three instruments, three jobs. Treating the cheapest one as the metric is how a system with steady thumbs-up rates degrades for a quarter without anybody noticing.