Weighted score fusion
Combining retriever scores, not ranks: normalise each list then blend, e.g. α·dense + (1−α)·BM25, with α tuned on labelled queries.
Different searches give scores on different scales: BM25 might say 18.4 while Cosine similarity says 0.71. Score fusion first rescales each list to a common range (for example 0 to 1), then mixes them with a weight you choose, such as 60% meaning-search and 40% keyword-search. Unlike Reciprocal Rank Fusion (RRF), it keeps the information about how much better one result is than the next.
Combining a maths mark out of 150 and an essay mark out of 20: you convert both to percentages first, then decide how much each counts.
Adding raw BM25 and cosine scores. BM25 runs 0–40 and cosine 0–1, so the sum is BM25 with noise.
Convex combination: s(d) = α·norm(s_dense(d)) + (1−α)·norm(s_bm25(d)). Normalisation choices: min-max per query over the retrieved set (simple, sensitive to one outlier), z-score (better when lists are long), or theoretical min-max (fixed bounds, e.g. cosine in [−1, 1]). Documents missing from one list get that list's minimum, not 0, or they are unfairly punished. Bruch, Gai and Ingber (2022) showed a tuned convex combination beats RRF in- and out-of-domain and is sample-efficient: α learned from tens to a couple of hundred queries transfers well. The weakness is coupling to the score distributions: swap the embedding model or BM25 parameters and α must be re-tuned, whereas RRF does not care. Learned fusion (a small LTR model over both scores plus features) is the next tier and needs thousands of labels.
Min-max normalise each list per query over its top 100; missing docs take that list's min; α from a 0.0–1.0 sweep in 0.1 steps on the golden set, picked by nDCG@10 with a bootstrap CI. Elasticsearch/OpenSearch linear retriever or OpenSearch normalization processor rather than app code. Store α in config with the embedding model version it was tuned for.
E-commerce catalog, 2M products, 300 labelled queries. RRF: nDCG@10 = 0.48. Min-max convex combination swept α from 0 to 1 in 0.1 steps: peak 0.52 at α = 0.6, with a 95% bootstrap CI of ±0.015, so the 4-point win is real. Short head queries ('running shoes') prefer α ≈ 0.7; part-number queries prefer α ≈ 0.1, so the team ships α = 0.6 with a code-shaped-query override to 0.1.
Say: start with RRF, move to a convex combination once you have 100+ labelled queries and a stable embedding model, and re-tune α in the same PR as any model swap. The follow-up is 'what breaks?': min-max on a 3-item list is unstable, so require a minimum list depth, and missing documents must take the list minimum.
RRF (ranks)
- No labels needed
- Immune to scale change
- Ignores score gaps
- k = 60 default
Convex combination
- Needs ~100 labels
- Re-tune on model swap
- Uses score gaps
- +1–3 nDCG when tuned
Numbers worth knowing
- Labels needed to tune α
- 50–200 queriessingle global α; per-class α needs ~50 per class
- Typical tuned α
- 0.5–0.7 on densenatural-language queries; 0.1–0.3 for identifier-heavy queries
- Gain over RRF
- +1 to +3 nDCG@10tuned in-domain; can be negative if α is stale
How to decide
| When | Do this |
|---|---|
| < 50 labelled queries | Stay on RRF |
| Clear query classes with different needs | Per-class α behind an intent router |
| Thousands of labels plus behavioural features (clicks, freshness) | Learning-to-rank (LightGBM LambdaMART) over both scores |
Cost maths
Compute is negligible. The cost is labels: 150 queries × ~10 judged results × 30 s each ≈ 12.5 hours of annotator time ≈ $750 at $60/hour (illustrative). That pays back if a 2-point nDCG gain moves a downstream metric you can price.
Operating it
- Embedding model or BM25 analyser changes: re-tune α before rollout
- Per-class nDCG spread > 10 points: split α by class
- Click logs exceed ~10k judged sessions: consider learning-to-rank
- Std-dev of normalised dense scores per query; alert on a > 30% shift (distribution change)
- Golden-set nDCG@10 of fused vs each single retriever; alert if fused falls below the better single one
- Config check: α's tuned-for model version equals the deployed embedding model
When it goes wrong
Hybrid search on an internal wiki got visibly worse the day after an embedding upgrade that had won offline
Weekly golden-set run: hybrid nDCG@10 fell 0.55 → 0.47 while dense-only rose
The new model's cosine scores clustered in 0.78–0.86; per-query min-max stretched tiny gaps into the full 0–1 range, overpowering BM25 at the old α = 0.6
Re-swept α (new optimum 0.4), switched to z-score normalisation, and tied α to the model version in config so CI fails if they diverge
Sources
- An Analysis of Fusion Functions for Hybrid Retrieval — Bruch, Gai & Ingber, 2022paper
- Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods — Cormack, Clarke & Büttcher, 2009paper
Check yourself
Three questions. Every option is explained once you answer.