Build Time and Query Time

A search system is a function from a query to a ranked list. The naive implementation scores every document against the query and sorts. It is correct, it is four lines, and it is unusable by five orders of magnitude.

Take a corpus of ten million documents and a scoring model that reads the query and one document together and returns a relevance score. A small cross-encoder does this in roughly ten milliseconds on a GPU. Ten million documents times ten milliseconds is twenty-eight hours per query.

The user is waiting for a few hundred milliseconds. The gap is five orders of magnitude, which means no amount of engineering closes it. You cannot cache your way out, or parallelize your way out at any sane cost. The architecture has to change, and every structure in a search system is a consequence of closing that gap.

1. The build/query split

Most of the work in a search system does not depend on the query.

Turning a document into tokens does not depend on the query. Counting how many documents contain each term does not. Computing a document’s embedding does not. Counting how often people click it does not. All of it can happen ahead of time, once, for the whole corpus, and be read cheaply later.

That is the split:

1. Build time runs on a schedule, has the whole corpus in view, and is measured in minutes to hours. It reads documents and writes artifacts.
2. Query time runs per request, has one query in view, and is measured in milliseconds. It reads artifacts and writes a ranked list.

The split is not an optimization layered onto a working system. It is the shape of the system. Asking which clock a piece of the stack is on determines what it is allowed to cost, what it is allowed to know, and how stale it is permitted to be.

Precomputation costs freshness. Anything computed at build time is as old as the last build, so a document edited five minutes ago is not in an index built last night. The query path cannot fix this, because noticing the edit would cost it time it does not have. The mitigations belong to the batch job: shorten the build interval, or apply changed documents incrementally between full rebuilds. Both are covered in Building the Index.

2. The two-scorer chain

Moving work off the query path is not enough on its own. Even with everything precomputed, the query path still has to consider ten million documents to rank them, and the twenty-eight-hour scorer is exactly the thing that produces good rankings.

The second move is to accept two different scorers.

1. A cheap scorer runs over the whole corpus and is wrong often, but wrong in a specific and tolerable way: it is bad at ordering and adequate at separating plausible from implausible. Term-overlap scoring is cheap because an inverted index means you never touch a document that shares no terms with the query. Vector similarity is cheap because approximate nearest-neighbour structures do sublinear work.
2. An expensive scorer runs over a few dozen documents and is much better at ordering.

Chain them and the arithmetic works. The cheap scorer reduces ten million to a few hundred in single-digit milliseconds. The expensive scorer orders those few hundred, or more often the top few dozen, in a hundred milliseconds. Total cost is dominated by a stage that runs on a fixed-size input, so it does not grow with the corpus.

This is the retrieve-then-rank funnel, the second structural fact about search after the build/query split. Retrieval optimizes recall: get the right documents into the candidate set, order roughly. Ranking optimizes precision: order the candidate set correctly. They are different jobs with different metrics, and conflating them is a reliable source of confused design discussions.

The funnel also sets an upper bound. A document that retrieval misses cannot be recovered by any amount of reranking. The expensive scorer only ever sees the candidate set. If the cheap scorer never surfaced the right answer, the good scorer never gets a chance to promote it, and every relevance metric measured downstream is measuring how well you ordered a set that did not contain the answer. Recall at the retrieval stage is the ceiling on everything.

3. The filtering stages, and the candidate margin they force

Between retrieval and the final list sit several stages, and they share one property.

Every one of them can remove documents. None of them can add documents.

The stages vary by system, and a representative set:

• Deduplication, because the same document can enter the candidate set more than once, especially when documents are split into chunks and several chunks of one document match.
• Collapsing, folding chunks back into the parent document so the user sees one result rather than four passages from the same page.
• Similarity floors, dropping anything scoring below a threshold on the grounds that a bad result is worse than a short list.
• Reranking cutoffs, keeping the top few dozen.

Ask retrieval for ten candidates, run them through these, and you get fewer than ten results. Not because anything failed. Because the pipeline is a sequence of filters and it was fed exactly as many items as the output needed.

There is no error. Every stage did its job. The user gets a short list, or an assistant reading those documents answers from thinner evidence than intended, and the only symptom is slightly worse output. Systems that appear to be underperforming on relevance are sometimes underperforming on candidate supply, and those have completely different fixes.

The correct shape is to size the candidate pool against the filtering, not against the output. If a ten-document result needs to survive dedup, collapsing and a floor, retrieval has to produce meaningfully more than ten. How much more is measurable: instrument what each stage drops and derive the multiplier from the observed loss.

One sharp edge specific to sharded indexes. Candidates are usually pulled per shard and merged, so a request for ten documents across four shards asks for three per shard. If the deduplication key is finer than the document, chunks rather than documents, then several candidates can collapse into one result and the pool shrinks below the request before filtering even starts. Requesting a per-shard count derived by dividing gives no margin, and a system that looks correct at every step returns short lists at the end.

4. The agreement contract between the clocks

Build time and query time are separate processes. They usually run on separate machines, on separate schedules, and often in separate languages, because the constraints differ: build time is a batch data job and gravitates toward the data-engineering stack, query time is a low-latency service and gravitates toward whatever the serving engine is written in.

But they must agree about three things:

1. Both must produce identical tokens from the same text.
2. Both must apply the same term-weighting formula.
3. Both must resolve to the same version of the same artifact.

When they disagree, nothing crashes. The query path looks up a key that build time never wrote, gets a miss, falls back to a default, and returns a plausible ranked list scored against weights that do not belong to it. There is no exception, no log line, no failed health check. The system is confidently wrong, and it stays that way until someone measures relevance carefully enough to notice it dropped.

The contract shows up in three places that look unrelated and are one problem:

1. Tokenizer parity between two runtimes.
2. Formula parity between the statistics job and the scorer.
3. Version pinning between an evaluation harness and a moving index alias.

I would put this first among the things a search system costs to build, above the ranking mathematics, which is settled.

The general defence is the same in each case. Make the agreement checkable rather than assumed. A parity test that runs sample input through both implementations and asserts identical output converts a silent production corruption into a failing build. Nothing else works reliably, because the failure produces no signal on its own.

A worked example

A concrete pipeline, to make the split and the funnel arithmetic real.

A help centre has 50,000 articles. Articles are long, so they are split into passages of roughly 200 words for retrieval, giving about 250,000 passages. A user asks a question and an assistant needs the ten most relevant articles to answer from, inside 400ms.

Build time, nightly:

1. Read 50,000 articles, split into 250,000 passages.
2. Analyze each passage into tokens. Write the inverted index.
3. Count, for each term, how many passages contain it. Write the term statistics.
4. Embed each passage. Write the vector index.
5. Aggregate a month of click logs per article. Write the evidence table.

Cost: roughly an hour. Nothing here is on the user’s clock.

Query time, per request, with a 400ms budget:

1. Analyze the query into tokens. Under 1ms.
2. Lexical retrieval, 100 candidates. About 15ms.
3. Semantic retrieval, 100 candidates. About 20ms.
4. Fuse the two lists into 150 unique passages. Under 1ms.
5. Rerank the top 50 with a cross-encoder. About 250ms, and this dominates.
6. Collapse passages to articles, apply the floor, take ten. Under 1ms.

Total around 290ms, inside budget, with the expensive stage on a fixed-size input.

Now the funnel arithmetic on step 6. Those 50 reranked passages are not 50 articles. Long articles contribute several passages each, so 50 passages might collapse to 30 articles. The similarity floor drops another handful. If eight survive, the assistant answers from eight documents while every component reports success.

Fixing it means reranking more passages, which costs latency directly. That is the trade the funnel imposes: the candidate multiplier and the latency budget are the same knob viewed from two ends, and the only way to set it is to measure what each stage actually drops rather than reasoning about what it should drop.