BM25
The default keyword ranking formula in Lucene, Elasticsearch, OpenSearch and Vespa: rewards rare terms, saturates repeats, normalises length.
BM25 ("Best Match 25") is the scoring formula most search engines use for keyword queries. A document earns points for each query word it contains; rare words earn more than common ones, repeating a word earns less and less extra, and very long documents are penalised so they do not win just by containing more words. It is the engine behind most Sparse retrieval.
Marking essays: using a rare, precise term scores well, using it 50 times does not score 50 times better, and a 40-page essay does not win just by being longer.
Adding raw BM25 scores to cosine similarities. BM25 is unbounded (5, 25, 60 depending on query length and IDF) while cosine sits in [−1, 1], so a weighted sum is dominated by whichever leg has the larger range. Fuse by rank with Reciprocal Rank Fusion (RRF), or min-max normalise per query in Weighted score fusion.
Score = Σ over query terms of IDF(t) × tf·(k1+1) / (tf + k1·(1 − b + b·dl/avgdl)). k1 sets term-frequency saturation: the per-term contribution approaches IDF·(k1+1) as tf grows. b sets length normalisation: 0 ignores length, 1 fully normalises. Lucene (and so Elasticsearch/OpenSearch) defaults to k1=1.2, b=0.75 and uses IDF = ln(1 + (N − n + 0.5)/(n + 0.5)), which is always positive. Vespa exposes the same parameters through its bm25 rank feature. Scores depend on per-shard statistics (N, n, avgdl), so the same document scores differently on different shards unless you use DFS query-then-fetch or have enough docs per shard for stats to converge. Tuning k1/b moves nDCG by 1–3 points; analyzers, field boosts and synonyms move it by 5–15.
Engine defaults k1=1.2, b=0.75; per-field similarity only after an offline eval shows a gain. Title field boost 2–3×, multi_match best_fields for prose, cross_fields for structured name/address fields.
Financial filings index, 2M 10-K sections, avgdl 400 tokens. Query goodwill impairment. 'impairment' is in 60k sections → IDF ≈ ln(1 + 1.94M/60k) ≈ 3.5. A 400-token section with tf=4: 3.5 × 4·2.2/(4 + 1.2) = 5.9. Same section with tf=40: 3.5 × 88/41.2 = 7.5 — 10× the repeats, 27% more score. A 4,000-token section with tf=4: denominator becomes 4 + 1.2·(0.25 + 0.75·10) = 13.3 → 2.3. Length normalisation cuts it by 60%.
When asked to tune lexical search, say you fix the analyzer and field boosts first and only then grid-search k1 ∈ [0.9, 2.0] and b ∈ [0.3, 0.9] on a labelled set: "BM25 is TF-IDF with saturation (k1) and length normalisation (b); I lower b for corpora where long documents really are more relevant, like statutes." Expect: can you threshold BM25 scores? No — they are unbounded and corpus-dependent.
Numbers worth knowing
- Lucene defaults
- k1 = 1.2, b = 0.75Elasticsearch, OpenSearch, Solr
- Saturation cap
- IDF × (k1 + 1)per term, as tf → ∞
- Gain from k1/b tuning
- +1–3 nDCG@10 pointsafter analyzers are fixed; analyzer/boost work is usually worth 5–15
How to decide
| When | Do this |
|---|---|
| Long documents are legitimately more relevant (statutes, manuals) | Lower b to 0.3–0.5 so length is penalised less. |
| Short, uniform fields (titles, product names) | Lower k1 toward 0.5–0.9; repeats carry little signal. |
| Combining with a dense leg | Fuse by rank with Reciprocal Rank Fusion (RRF) (k=60); do not add raw scores. |
Cost maths
No model cost. Tuning cost is the labelled set: 200 queries × 20 judged docs = 4,000 judgements; at ~30 s each that is ~33 annotator-hours, or ~$8 with an LLM judge at $3/$15 per M tokens (assume 1.5k tokens per judgement, illustrative Oct 2026 prices) plus a human audit of 10%.
Operating it
- Identical queries rank differently across shards — enable DFS or consolidate small shards
- Long-document bias complaints — tune b per field
- Recall ceiling reached on paraphrases — add SPLADE or a dense leg
- nDCG@10 on the lexical golden slice per release — alert on a drop > 2 points
- Score distribution drift per shard (median top-1 score) — alert if one shard deviates > 30%
When it goes wrong
In a legal-contracts search, a 2-page NDA template outranked the 60-page master agreement that actually defined the indemnity clause.
Lawyers' thumbs-down rate on 'definition' queries was 3× the baseline in weekly feedback review.
b=0.75 heavily penalised the long master agreements; the short template repeated 'indemnity' in boilerplate.
Indexed contracts at clause level instead of whole documents (see Chunking) and set b=0.4 on the clause-body field; nDCG@10 on the definition slice rose from 0.51 to 0.68.
Where it shows up
Sources
- The Probabilistic Relevance Framework: BM25 and Beyond — Robertson & Zaragoza, 2009paper
- Elasticsearch reference: Similarity module — Elastic, 2026docs
Check yourself
Three questions. Every option is explained once you answer.