Hybrid search and reranking: a field guide

Two tickets land on an internal developer-search tool in the same week. One engineer searched ERR_CONN_RESET payments-gateway and got a page of generic networking articles; the runbook that names that exact error string ranked 31st. Another asked “why does checkout drop connections under load?” and got nothing useful, though the same runbook answers it in different words. The first is a dense-retrieval failure, the second a sparse one, and the standard fix for both is to run the two retrievers side by side and fuse their results.

Sparse retrieval, almost always BM25 in Elasticsearch, OpenSearch or Lucene, scores shared terms weighted by rarity. It nails identifiers, error codes, SKUs, function names and rare jargon, and it has no idea that “drop connections” and “connection reset” are related.

Dense retrieval embeds query and document separately with a bi-encoder and compares vectors. It handles paraphrase, and cross-lingual queries with a multilingual model. It is unreliable on exact strings: ERR-4012 and ERR-4021 tokenise into near-identical pieces and land close together, and an internal acronym the model never saw in training has no meaningful position at all.

A BM25 score of 14.2 and a cosine of 0.83 live on different scales whose distributions shift per query. Normalising and adding them, score fusion, works only with per-corpus calibration that breaks every time either retriever changes. Reciprocal Rank Fusion ignores scores and uses positions: each document gets Σ 1/(k + rank) over the lists it appears in, with k = 60 from Cormack et al. (2009).

With k = 60, appearing in both lists is worth ~0.032 and appearing in one is worth ~0.016, so consensus roughly doubles a document's score, while rank 1 versus rank 4 within one list differs by only 5%. Lower k (10–20) makes top positions matter more, which helps when one retriever is clearly stronger on your traffic. Treat it as a hyperparameter only once you have a golden set to tune it on; Elasticsearch, OpenSearch, Qdrant and Weaviate all ship RRF natively, so in production you mostly set k and the per-list depth.

SPLADE (Formal et al., 2021) is learned sparse retrieval. A BERT-style model outputs a weight for every term in its ~30,522-token vocabulary, including terms not in the text, so “connection reset” can also light up drop and socket. The result is still a sparse vector served from an ordinary inverted index, which buys most of dense retrieval's paraphrase handling while keeping BM25-style exact matching and infrastructure. The cost is a GPU-friendly encoder at indexing time and larger postings lists than BM25.

ColBERT (Khattab & Zaharia, 2020) is late interaction. Instead of one vector per chunk it stores one small vector per token, and scores a query by summing, for each query token, its best match among the document's tokens (MaxSim). It approaches cross-encoder quality at retrieval-time speed, but storage is the bill: 1M chunks × 200 tokens is 200M vectors, about 51 GB at 128 dimensions in FP16. ColBERTv2's residual compression brings each vector to roughly 20–36 bytes, around 4–7 GB. Reach for it when a cross-encoder's latency is unaffordable but bi-encoder quality is not enough.

Fusion decides which candidates survive. It never reads query and document together. A reranker, usually a cross-encoder, does: it concatenates query and passage, runs a full transformer over the pair and outputs one relevance score, so it can see that a passage mentions payments-gateway but is about deploys, not connection resets (Nogueira & Cho, 2019, used exactly this to beat the previous state of the art on MS MARCO by 27% relative MRR@10).

Nothing about a cross-encoder can be precomputed: every candidate is a fresh forward pass per query. The latency numbers below are back-of-envelope estimates from parameter counts, not benchmarks. Assumptions: 100 candidates, 256-token query-plus-passage pairs, batched, FP16 on one L4 or A10G-class GPU, FP32 on an 8-vCPU x86 server. Measure on your own hardware before committing.

Two operational conclusions follow. MiniLM-L6 is cheap enough to run on CPU next to your search service for 20–50 candidates. A 568M multilingual reranker is a GPU service, with its own capacity planning, batching and p99 budget; on CPU it will dominate your end-to-end latency. Hosted rerank APIs move that cost to a per-request bill and a network hop of 50–150 ms.

Decide with two numbers from your golden set: recall@50 of the fused candidates and recall@5 of what reaches the prompt.

• recall@50 ≥ 0.9, recall@5 ≤ 0.7: the right chunk is in the pool but buried. This is exactly the gap a reranker closes. Add one.
• recall@50 < 0.8: the right chunk is not in the pool; no reranker can promote what it never sees. Fix chunking, add the missing retriever, or try query rewriting first.
• recall@5 already ≥ 0.9: little headroom. Spend the latency elsewhere.
• Tight p95 budget: rerank 20 instead of 50, use MiniLM-class models, or skip reranking when the top fused result leads the second by a wide RRF margin.

Track nDCG@10 or MRR alongside recall, since a reranker's job is ordering. If the metric moves by less than run-to-run noise on a 200-question set, the reranker is latency without benefit.

BM25 and a dense retriever in parallel, 50 candidates each; RRF with k = 60; a MiniLM-class cross-encoder on the top 30 if the recall gap above exists; top 5 into context assembly. Put each stage behind a feature flag and log per-stage latency, so every layer has to keep earning its place on the eval and on the p95 dashboard.