MRR (mean reciprocal rank)
Mean Reciprocal Rank: average of 1/rank of the first relevant result per query. Answers 'how far down is the first good hit?'
For each question, find where the first useful result appears. If it is first, score 1; second, 1/2; third, 1/3; not found, 0. MRR is the average of those scores. It is the natural metric when one good result is all you need.
Rating a waiter by how many tries it takes to bring the dish you ordered: first try is full marks, second try half.
Using MRR for RAG that needs several chunks. A system that nails one chunk and misses the other three scores perfectly.
MRR = (1/|Q|) Σ 1 / rank_q, where rank_q is the position of the first relevant item; 0 if none in the cutoff (MRR@10 sets anything beyond 10 to 0). MS MARCO passage ranking uses MRR@10 as its official metric, which is why so many rerankers report it. MRR is steep: rank 1 → 2 costs 0.5, rank 9 → 10 costs 0.011, so it reflects whether the top result is right. It ignores every relevant item after the first, so it is wrong for multi-document synthesis; use nDCG or Recall@k there. It is also an average of a bounded, skewed distribution, so report it with the hit rate@1 and the share of queries with no hit, which tell you more than the mean alone.
MRR@10 for single-target retrieval (FAQ match, entity lookup, code definition), reported with hit@1, hit@10 and miss rate, per slice, with a paired bootstrap CI when comparing systems.
Code search over an SDK, four queries. 'parse ISO-8601 duration': first relevant at rank 1 → 1.0. 'retry with jitter helper': rank 3 → 0.333. 'deprecated v1 auth client': not in top 10 → 0. 'stream upload large file': rank 2 → 0.5. MRR@10 = (1 + 0.333 + 0 + 0.5) / 4 = 0.458. Hit rate@1 = 25%, no-hit = 25%: the no-hit query is the one to investigate first.
Say: I use MRR@10 when there is one correct target, such as a function definition, a FAQ entry or an entity, and nDCG when several documents contribute. I always show it with hit@1 and miss rate. Expect: 'MRR went up but users are unhappy': check whether misses went up too; MRR can rise on easy queries while hard ones fall out of the top 10.
| Query | First relevant rank | Reciprocal rank |
|---|---|---|
| parse ISO-8601 duration | 1 | 1.000 |
| retry with jitter helper | 3 | 0.333 |
| deprecated v1 auth client | none in 10 | 0.000 |
| stream upload large file | 2 | 0.500 |
| MRR@10 | 0.458 |
Numbers worth knowing
- MS MARCO dev MRR@10
- ~0.18 BM25, ~0.39–0.40 BERT-class cross-encoderpassage ranking; approximate published figures
- Rank 1 → 2 penalty
- 0.5 per queryvs 0.011 for rank 9 → 10
- Useful companion stats
- hit@1, miss ratealways report with the mean
How to decide
| When | Do this |
|---|---|
| Exactly one right answer (navigational, FAQ, entity) | MRR@10 |
| Answer needs several documents | nDCG@10 and recall@k instead |
| UI shows only one result (autocomplete, 'I'm feeling lucky') | hit@1, which MRR approximates |
Cost maths
Cheapest ranking metric to label: binary relevance, and you can stop judging at the first relevant result. 400 queries × ~4 docs judged × 20 s ≈ 9 hours ≈ $540 at $60/h.
Operating it
- Product moves from single-answer lookup to multi-document answers: switch to nDCG
- Miss rate > 15%: first-stage recall problem
- MRR and hit@1 diverge: the middle of the ranking is changing
- MRR@10, hit@1 and miss rate nightly; alert if miss rate rises > 2 points
- Per-slice MRR (head vs tail queries); alert on a tail drop > 0.05
When it goes wrong
A FAQ matcher's MRR rose from 0.71 to 0.76 after retraining, yet escalations to human agents went up
Escalation dashboard; per-query diff showed misses rose from 9% to 14%
The new model improved already-answered easy queries (rank 2 → 1) while pushing long-tail queries out of the top 10; the mean hid it
Release gate now requires miss rate not to rise by > 1 point in addition to MRR, and long-tail queries got their own slice
Related terms
Where it shows up
Sources
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset — Bajaj et al., 2016paper
- Introduction to Information Retrieval (ch. 8, Evaluation) — Manning, Raghavan & Schütze, 2008book
Check yourself
Three questions. Every option is explained once you answer.