Ranking on Evidence

Retrieval and ranking score the relationship between a query and a document: how well the words match, how close the meanings sit. That relationship does not settle every ordering, because two documents can match a query equally well while one is the canonical answer and the other a stale edge case that happens to use the same words.

Distinguishing them requires evidence about the documents themselves, independent of the query. The obvious source is behaviour: which documents people actually click, read, and stop searching after.

1. Query-independent evidence

A query-independent feature is a property of a document that is true before any query arrives: click count, dwell time, recency, whether it is canonical or a duplicate, editorial quality, authority.

The category matters because of the clocks. Query-independent means computable at build time, precomputed once per document, and read at query time as a lookup. That is what makes these features affordable at all. A feature requiring per-query computation over the candidate set competes directly with the reranker for the latency budget; a feature that is a column on the document costs nothing.

The dominant one is engagement, and the reason is that it is a direct signal of usefulness rather than a proxy for it. Matching words or meanings is an inference about whether a document answers a question. A click is a user choosing that document, and repeated clicks across many users is evidence no textual analysis produces.

2. Turning interaction logs into a feature

Engagement arrives as raw interaction logs and needs real work before it is a feature.

• Aggregation window. Counts over what period? Short windows are fresh and noisy, long windows are stable and slow to reflect that a document became obsolete. Systems usually carry several, a 30-day and a 90-day count, and let ranking choose.
• Normalization. Raw click counts measure exposure as much as quality. A document ranked first for a popular query accumulates clicks because it is ranked first: engagement features trained on the current ranking reinforce the current ranking. The document at position one gets clicks, the clicks confirm it belongs at position one, and a genuinely better document at position eight never accumulates the evidence that would promote it. Serious treatments handle this with position-bias correction or by dividing clicks by impressions rather than counting them raw.
• Identity. Clicks arrive keyed by URL, documents are keyed by document id, and one document often has several URLs. Getting this join wrong silently splits a document’s evidence across keys, which is the same failure shape as the analyzer parity problem in a different place.
• Definition parity. If an offline analysis measured a lift using one click definition, the production feature must use that same definition. Reusing the analysis query rather than writing a near-copy is the discipline that keeps the shipped feature the one that was measured.

3. Learning to rank

One feature can be added to a score by hand. A dozen cannot, and hand-tuned weights across many features stop being defensible quickly.

Learning to rank replaces the hand-tuning with a model trained on judged data. Assemble, for a set of queries, the candidate documents and a relevance label per document, either human-judged or click-derived. Represent each query-document pair as a feature vector: the lexical score, the semantic score, the engagement count, freshness, and so on. Train a model to order documents by relevance.

Two consequences follow.

1. The retrieval scores become features rather than the answer. BM25 and cosine similarity enter as inputs alongside everything else, and the model learns how much to trust each in context. This is why the reranking stage is the natural place for learning to rank: it operates on a small candidate set where per-document feature computation is affordable.
2. The label is the hard part, not the model. Gradient-boosted trees on ranking objectives are a solved product. Getting labels that mean what you need them to mean is where the difficulty lives, and click-derived labels carry exactly the position bias above: they record what the current ranking showed people. A model trained on biased labels learns the bias faithfully.

4. Engagement as a head-only feature

Engagement is distributed across a corpus with extreme concentration, and that distribution is what bounds the whole approach. Measured over one month of real traffic against a corpus of roughly 11,000 documents:

• 2,273,544 clicks over 7,027 documents that the corpus holds.
• Coverage is 64%, so more than a third of documents received no clicks at all.
• Of covered documents, 44.1% had fewer than ten clicks in the month, leaving roughly 36% of the corpus with ten or more clicks a month.
• Median 13 clicks, p90 464, p99 4,697, maximum 275,201.
• The top 1% of documents take 48.9% of all clicks. The top 10% take 86.5%. The bottom half take 0.6%.

Engagement is a head feature. For the head of the corpus it is strong, dense and discriminative. For the tail, which is most of the corpus by document count, the raw count is either zero or a number too small to separate anything, and a feature that cannot distinguish two documents contributes nothing to ordering them.

Three consequences follow.

1. Expected lift is bounded by coverage, not by feature quality. If the feature is informative on a third of the corpus, no amount of modelling improves it on the rest. Offline evaluations run over popular queries look much better than production, because popular queries hit the covered head by construction.
2. Raw counts are the wrong representation. A feature whose values span zero to 275,201 with a median of 13 needs a transform before a model can use it sensibly: a logarithm, a smoothed rate, or a Bayesian estimate shrinking sparse documents toward a prior. The last is the principled version, because it encodes that three clicks on a rarely-seen document is weak evidence rather than a small number.
3. Freshness matters less than it appears. A head document’s engagement changes slowly because it already has a large count, and a tail document’s is noise regardless. This bears directly on a common design question, which is whether features need their own update cadence separate from the document build: if features ride the document hash and only refresh when a document’s text changes, a click-only update does not propagate. For a distribution this concentrated, that staleness costs less than it seems, and the honest answer is that it should be measured rather than assumed in either direction.

5. The size of the effect, and the pipeline that outlasts it

Engagement features in a system that already has good hybrid retrieval and reranking produce single-digit percentage relevance improvements, not step changes.

That is worth having. It is also small enough that measuring it is genuinely hard, which is the subject of the next section, and small enough that a system reporting a large gain from adding click counts is more likely to have a measurement problem than a breakthrough.

The pipeline outlasts the first feature that runs through it. Building the path from behavioural log to build-time aggregate to per-document store to ranking input means the second feature and the tenth cost almost nothing, and the shape that buys this is a single extensible field rather than a column per feature: one dictionary on the document, keys mapping to columns downstream, so adding a feature needs no schema change in three separate systems.

A worked example

An offline evaluation predicts one thing and the online experiment measures another, and both numbers are real. This is the ordinary case rather than a failure, and reading it correctly is the skill.

Offline. Sweeping engagement weight over 80,275 candidate rows covering 5,000 conversations predicted +6.5% relative on average relevance at the top five, with a guardrail showing 5.5 percentage points fewer irrelevant documents in the top five. Gains saturated and then reversed past the best weight, which is the expected shape: some engagement weighting helps, too much overrides relevance and promotes popular documents that do not answer the question.

Online. The same feature ran as an arm of a live experiment over one week, 6,369 subjects in that arm, and came in at −0.29% relative on the decision metric.

The naive reading is that offline lied. The correct reading is in the third number: the experiment’s minimum detectable effect was 2.80% relative at that sample size. The measured −0.29% is an order of magnitude inside the resolution of the instrument. The run cannot distinguish −0.29% from +2%, and detecting a 1% relative effect would have required roughly 60 days at that traffic.

So the online result is not evidence against the feature. It is not evidence for it either. It is not evidence. An experiment whose minimum detectable effect exceeds the effect anyone expected was never going to answer the question, and the time to notice that is before running it, by computing what the experiment could resolve and comparing it to what the offline work predicted.

The distributional numbers above explain why the true effect is plausibly small: the feature is informative on roughly a third of the corpus, so a 6.5% lift measured on a candidate sweep weighted toward covered documents becomes something much smaller when averaged over all production traffic.