Fusion and Reranking

A serious search system runs two retrievers over every query. One matches literal words, which wins on error codes and product names. One matches meaning, which wins when the user describes a problem in words the document never uses. They fail on different queries, which is why both are there.

So two ranked lists come back and the user gets one. Everything between those facts is this page: first merging the lists, then spending the query budget’s remaining milliseconds improving the order at the top.

1. The scale mismatch between the two arms

The obvious merge is to add each document’s scores from both arms and sort. It fails, and the same mistake appears whenever two systems’ outputs get combined.

The two scores are not on the same scale, and neither has a fixed range. A BM25 score is an unbounded sum of per-term weights: it can be 4 on one query and 40 on another, and the value depends on how many query terms matched, how rare they were, and how long the document is. A cosine similarity is bounded in [−1, 1] and clusters tightly, often between 0.6 and 0.9 for anything plausible.

Add them and BM25 dominates whenever its scale runs large, which is a property of the query rather than of relevance. The semantic arm’s best result loses to the lexical arm’s fourth-best, not because it is worse but because its numbers are smaller.

Normalizing each list to [0, 1] before adding is better and still fragile. Min-max normalization is defined by the extremes of the list, so a single outlier compresses everything else into a narrow band, and a list where every result is mediocre gets its top result promoted to 1.0 exactly as if it were excellent. Normalization makes the ranges match without making the scores mean the same thing.

2. Reciprocal rank fusion

The standard solution discards the scores and keeps only the positions.

rrf_score(d) = Σ  1 / (k + rank(d, L))
              L

summed over every list L in which document d appears, where rank is its 1-based position and k is a constant conventionally set to 60.

The reasoning is that rank position is the one quantity both retrievers produce that means the same thing. Being first in the lexical list and first in the semantic list are directly comparable statements, even though a BM25 score of 40 and a cosine of 0.91 are not. Reading only rank makes the scale mismatch irrelevant by refusing to look at the scales.

The 1/(k + rank) shape gives a large boost to the top of each list, decaying quickly. The constant k controls how quickly: small k makes first place dramatically better than third, large k flattens the difference so that appearing in both lists matters more than placing highly in one. At k = 60 the difference between rank 1 and rank 2 is modest, which is deliberate, since it favours documents both retrievers liked over documents one retriever loved.

That is the actual behaviour to keep in mind: RRF rewards agreement. A document ranked third by both arms often outscores one ranked first by a single arm and missing from the other.

What it gives up. Rank position ignores margin. If the top lexical result scores 40 and the second scores 4, that gap says the first result is overwhelmingly better, and RRF sees only “first, second.” A retriever that is confident and a retriever that is guessing produce identical fused contributions. In systems where score calibration is genuinely trustworthy, a learned combination can beat RRF. The reason RRF is the default is that calibration usually is not trustworthy, and RRF requires no tuning, no training data, and no per-query normalization to work acceptably.

3. The cross-encoder

Fusion produces a candidate list quickly and roughly. The ordering at the top is what the user experiences, and rough is not good enough there.

Both retrievers share a limitation that follows from how a search system is built. Almost everything they read was precomputed by a batch job, because a few hundred milliseconds is not enough time to process a corpus. So they compare representations built independently of each other. The document’s vector was computed ahead of time, before the query existed. The term statistics were computed the same way. Retrieval is a comparison between a query representation and a document representation that were never in the same room.

A cross-encoder removes that constraint. It takes the query and the document text together as one input and runs attention across both, so every token of the query can attend to every token of the document. It produces a relevance score directly rather than a point in a space.

The gain is that the document representation is built conditional on the query. A bi-encoder must compress a document into one vector before knowing what will be asked of it, so it has to guess which aspects matter. A cross-encoder never commits early.

Where this shows up concretely is in queries whose meaning depends on how terms attach to each other. Consider a query for a small fast car against two documents:

Both documents contain both terms, so any method that reduces a document to a bag of content scores them the same. The difference is which noun the adjectives modify, and that is a relationship between positions that only survives if the model can attend across query and document together.

4. Rerank depth

The cross-encoder cannot be the only scorer, because its work cannot be precomputed. There is no document vector to build ahead of time because the score depends on the query, so every document scored costs a full forward pass at query time.

That turns rerank depth into arithmetic. A reranker has some slice of the latency budget, the model costs some milliseconds per document, and the quotient is how many documents can be reranked. At roughly 250ms of budget and a small cross-encoder, that lands in the tens.

The depth choice is a genuine trade with a hard bound above it:

Rerank deeper and more of the true answers are inside the window the good model can reorder, at higher latency. Rerank shallower and you are faster but the good model never sees documents that retrieval ranked poorly.

Reranking precision is bounded above by retrieval recall at the rerank depth. If the right document is at position 80 and you rerank the top 50, no reranker quality helps. This is the retrieve-then-rank ceiling stated at the stage where it actually binds, and it is the reason a system with a disappointing reranker is sometimes a system with a retrieval problem.

There is also a model-size version of the same trade. A larger cross-encoder orders better per document and is slower per document, so it reranks fewer documents in the same budget. Whether the better ordering over 30 documents beats a weaker ordering over 80 is corpus-specific and measurable.

One fork deserves stating plainly because it is usually skipped: whether the reranker earns its place at all. It costs latency on every query, and it only pays if the fused ordering is meaningfully wrong. On corpora where retrieval already orders well, a cross-encoder is cost with no benefit. The honest version is to measure the relevance lift against the latency and keep it if the lift is real, rather than including a reranker because the architecture diagram has one.

A worked example

Two retrievers return the same six documents in different orders. The lexical scores run on a scale roughly ten times the semantic ones, which is the ordinary case and the reason naive addition breaks.

The interesting document is B: the semantic arm’s top result, ranked fifth by the lexical arm. It is the document that matches by meaning without sharing terms, which is precisely what the semantic arm is in the system to catch. Where each fusion method places it:

The full RRF result, at k = 60:

A and B finish first and second: each is one arm’s top result, and RRF gives both the same reciprocal boost regardless of the underlying scores. C, ranked third by both arms and top of neither, lands third overall, which is the agreement reward working as designed. D and E tie exactly, being mirror images of each other at ranks 2 and 6.

Naive summation put B fifth. It would have been dropped by any cutoff below five, and the semantic arm would have contributed nothing to the final list despite having found the right document first.