Measuring a Relevance Change

Every change to a search system is a claim that ranking got better, and relevance is not directly observable. There is no counter to read. Whether a result was useful is a fact about a person, and every method for measuring it is an instrument with a resolution limit.

The measurement is usually the binding constraint, not the change. The retrieval improvements available to a mature system are worth low single-digit percentages, which is small enough that most experiments cannot see them at all, and an experiment that cannot see the effect it was built to find will confidently return nothing.

1. Offline metrics and the labels they need

Offline evaluation compares rankings against known-relevant answers, needing a set of queries, a set of documents, and relevance labels.

The metrics are standard, and their differences matter less than their inputs.

• Precision at k is the fraction of the top k that is relevant. Simple, and blind to ordering inside the window and to everything below it.
• Recall at k is the fraction of all relevant documents that appear in the top k. The retrieval-stage metric, because retrieval’s job is getting candidates into the set.
• Mean reciprocal rank is the reciprocal of the position of the first relevant result, averaged over queries. The right metric when the user needs one answer.
• Normalized discounted cumulative gain discounts each result’s contribution by its position and normalizes against the best possible ordering. Handles graded relevance and full orderings, and it is the default in most serious evaluation.

The metric choice is not where the difficulty is. The labels are. Three sources, in descending cost:

• Human judgment is the gold standard: judges rate documents against queries on a graded scale. Expensive, slow, and inconsistent between judges, which is why judge agreement is itself measured.
• Click-derived labels are cheap and abundant and carry position bias: people click what was ranked highly, so the labels measure what the current ranking showed them. A system evaluated on click labels is partly being evaluated on how closely it reproduces itself.
• Model judgment has become standard: a language model reads a query and a result and scores it. Cheap enough to run at thousands of judgments, consistent in a way humans are not, and carrying its own biases. Its resolution is the constraint that actually bites, and it is the next section.

2. Judge resolution

The standard response to a noisy measurement is more samples. It is correct for sampling noise, and it does nothing for the limit that actually stops offline evaluation, which is how finely the judge can discriminate at all.

Consider replaying the same conversations through two retrieval configurations, having a model judge each finished result, and comparing. Real numbers from such a run: 1,298 paired comparisons, and 833 of them scored identically on both arms.

Sixty-four percent ties. Not close scores. The same score.

That is the judge failing to distinguish the two configurations at this effect size. And crucially, a tie contributes nothing to the estimated difference no matter how many of them you collect. More samples buy more ties in the same proportion. The confidence interval narrows slowly because the informative fraction is small, and the underlying resolution never improves.

Sample size limits precision. Instrument resolution limits what precision is available at all. Reporting a null result without checking the tie rate is reporting that your instrument returned nothing, which is not the same as the change doing nothing.

Signs the judge rather than the sample is the limit:

• A high exact-tie rate between arms.
• Effect estimates far inside the interval width.
• Required sample sizes that exceed what is runnable. On that run, detecting the observed effect would have needed roughly 6,000 pairs against runtime and memory ceilings that made it infeasible.

When the instrument is the limit, adding samples is the wrong move. The moves that work are a more sensitive judge, a metric closer to the change, or measuring behaviour instead of judgment.

3. Pairing and exposure conditioning

Pairing. Run both configurations over the same inputs and compare per-input differences rather than group means. This removes between-input variance, which usually dominates: some queries are simply harder than others, and that variation swamps a small treatment effect when comparing independent groups.

Measured on the same run, pairing was worth about a quarter of the precision: paired +0.023 with interval [−0.012, +0.057] against unpaired +0.017 with [−0.029, +0.062].

Pairing has a cost worth knowing. It requires both arms to have produced a comparable result for the same input, so any input where one arm did not is dropped, which shrinks the usable sample. Whether pairing wins depends on whether the variance it removes exceeds the sample it discards, and that is measurable rather than a matter of preference.

Conditioning on exposure. If a change only affects requests where a particular path runs, then requests where it does not run are pure noise: both arms do exactly the same thing, contributing a guaranteed zero difference plus measurement variance.

If a change affects 8.6% of traffic, a random sample of 2,000 yields about 165 informative comparisons, and the required sample scales with the square of the inverse rate. Selecting inputs where the relevant path fires took one real evaluation from 165 usable pairs to 1,329 on a comparable budget, an eight-fold increase in information for the same compute, with the exposure rate going from 8.6% to 58.3%.

The correctness condition is what makes this legitimate rather than cheating. The selection must be on a pre-treatment property: something true before the change was applied, and unaffected by which arm an input lands in. Selecting on whether the original production transcript used the path is pre-treatment. Selecting on whether the path fired during the experiment is post-treatment and biases the comparison, because the treatment can change which inputs qualify.

The verification is direct: check that the rate is the same in both arms. In that run the path fired 0.714 times per conversation in one arm and 0.716 in the other, which is what licenses conditioning on it.

4. The minimum detectable effect

Offline evaluation measures whether a judge thinks results improved. Online experiments measure whether users behaved differently, which is the question that actually decides a launch. Both are sized by one number, computed before the experiment runs.

The minimum detectable effect is the smallest true effect the experiment could reliably detect, given the sample it will collect, the variance of the metric, and the power required. It is a property of the design, not of the result.

Comparing the minimum detectable effect to the effect you expect resolves most experimental disappointment in advance. A worked case: a click-count ranking feature predicted at +6.5% offline, measured at −0.29% online, against a minimum detectable effect of 2.80%. That experiment could never have detected an effect of the size anyone expected. The result is not a null finding, it is an experiment that returned nothing because it was not sized to return anything.

That check costs ten minutes at design time. A retrieval improvement worth 1% relative in a system with typical variance needs a sample most teams do not have. Knowing that beforehand changes what you build: you pick a more sensitive metric, condition on exposure, or accept that the change ships on judgment rather than on measurement.

5. The honest report of a null result

Most retrieval changes measure flat, and reading that correctly means refusing two symmetrical errors.

1. An interval containing zero does not mean the change did nothing. It means the data are consistent with a range that includes zero. An interval of [−0.012, +0.057] is also consistent with a meaningful positive effect, and reporting it as “no effect” throws away the asymmetry.
2. A directional pattern is not a result. Every metric leaning positive is weak evidence rather than a finding, and it is the shape that survives motivated reading. It counts for something only when it reproduces independently and when the metric that moved most is the one that should move if the mechanism is real. On the hybrid run, grounding leaned hardest, and grounding is the metric most directly downstream of retrieval quality. That coherence is reportable and it is not a positive result.

What a careful null earns is the right to a different decision. A well-bounded negative with no regression, a plausible mechanism, an identified measurement limit, and a bounded downside is a defensible case for promoting to an online test where the instrument is user behaviour rather than a judge. A rigorous null buys more than an overclaimed positive, because it tells the next person what the instrument can and cannot see, and that information is durable in a way a point estimate is not.

The honest report of a flat result names four things:

1. The estimate with its interval.
2. The tie rate, or whatever else bounds the instrument.
3. The sample that would have been needed.
4. Whether the mechanism remains plausible.

Anything less is either an overclaim or an underclaim, and both cost the same in the end.

A worked example

A full offline comparison of two retrieval configurations, reported the way it should be.

The setup: replay 2,500 production conversations through both configurations, with the same simulated user driving both so the only difference is which retrieval path ran, and a model judge scoring each finished conversation on four dimensions. Conversation ids were pre-selected from production logs where the original transcript used the retrieval path, which is the pre-treatment conditioning described above.

Paired per-conversation deltas over the 1,298 pairs where both arms fired and scored, with bootstrap 95% intervals over 4,000 draws:

Wins 246, losses 219, ties 833.

Every interval contains zero, so this does not show the treatment is better. That is the first sentence of the report and it is not negotiable.

What can also be said, and should be: the interval is 2.9 times narrower than an earlier random-sample run, so the design change worked. The sign pattern reproduced independently across two runs. Grounding leans hardest and is the metric downstream of retrieval quality. Execution is flat, so nothing regressed. The retrieval path fired at essentially identical rates in both arms, 0.714 against 0.716, which is what makes the conditioning valid.

And the finding that matters most for what happens next: 833 exact ties out of 1,298. The judge is not separating these configurations at this effect size. Detecting the observed +0.023 would need roughly 6,000 pairs, past the runtime and memory ceilings of the harness. The limit is the instrument, not the sample, so a third offline run at higher sample would spend a week to produce the same interval slightly narrower.

The decision that follows is not “the change failed.” It is that offline measurement is exhausted, and the question moves to an online test where the measurement is user behaviour and the downside is bounded by the fact that nothing regressed.