nDCG
Normalised Discounted Cumulative Gain: graded-relevance ranking score that rewards the best results near the top. 1.0 = ideal order.
nDCG scores a ranked list when some results are more relevant than others (say 3 = perfect, 2 = good, 1 = partial, 0 = useless). Each result's score is shrunk the further down the list it sits, the shrunk scores are added up, and the total is divided by the best possible total. 1.0 means the best items are exactly where they should be.
Judging a playlist where great songs score more than good ones, and a great song at track 1 counts more than the same song at track 9.
Comparing nDCG across datasets or gain conventions. 0.45 on a hard BEIR set can be better than 0.80 on an easy internal set; only compare systems on the same queries and labels.
DCG@k = Σ_{i=1..k} gain(rel_i) / log2(i + 1). Two gain conventions exist: linear (gain = rel, used by trec_eval and scikit-learn's ndcg_score) and exponential (gain = 2^rel − 1, from Burges et al. and common in web search), which rewards perfect results much more. IDCG@k is the DCG of the ideal ordering of all judged items; nDCG@k = DCG@k / IDCG@k. Report which gain you used; numbers are not comparable across conventions. nDCG@10 is the headline metric on BEIR and MTEB retrieval tasks. It needs graded judgments, which cost more than binary ones, and it is averaged over queries, so a 0.02 change can be noise: use paired bootstrap or a paired t-test on per-query scores. Queries with no relevant judged documents have IDCG = 0 and must be excluded or set to 0 consistently.
nDCG@10 with graded labels (0–3), linear gain unless you have a reason for exponential, computed with ranx or pytrec_eval; per-slice reporting; paired bootstrap (10k resamples) for system comparisons.
Financial-filings search for 'goodwill impairment 2025 reporting unit'. Judged grades: d2 = 3, d4 = 2, d9 = 1, others 0. System ranks d7, d2, d9, d4, d1. Linear gain DCG@5 = 0/1 + 3/1.585 + 1/2 + 2/2.322 + 0/2.585 = 0 + 1.893 + 0.500 + 0.861 + 0 = 3.25. Ideal order d2, d4, d9: IDCG@5 = 3 + 2/1.585 + 1/2 = 4.76. nDCG@5 = 3.25 / 4.76 = 0.68. Moving d2 to rank 1 alone gives 0.89.
Say: nDCG@10 with graded labels is my primary ranking metric because it rewards putting the best evidence first, which is what the prompt budget needs; I pair it with recall@100 for the first stage and report paired-bootstrap CIs. Expect: 'linear or exponential gain?' Exponential if a perfect document is worth much more than a partial one (single-answer Q&A); linear if partials are genuinely useful (research synthesis).
| Rank | Doc | Grade | log2(i+1) | Gain / discount |
|---|---|---|---|---|
| 1 | d7 | 0 | 1.00 | 0.00 |
| 2 | d2 | 3 | 1.58 | 1.89 |
| 3 | d9 | 1 | 2.00 | 0.50 |
| 4 | d4 | 2 | 2.32 | 0.86 |
| 5 | d1 | 0 | 2.58 | 0.00 |
| DCG@5 | 3.25 | |||
| IDCG@5 | d2, d4, d9 | 4.76 | ||
| nDCG@5 | 0.68 |
Numbers worth knowing
- BEIR average nDCG@10
- ~0.42 BM25, ~0.50–0.55 strong dense + rerankzero-shot across BEIR datasets; approximate, varies by subset
- Detectable difference
- ≈ 0.02–0.03paired test, ~300–500 queries, typical per-query variance
- Graded labelling cost
- ~1.5× binaryper judged document, trained annotators
How to decide
| When | Do this |
|---|---|
| One right answer per query | MRR or hit rate@k is simpler and enough |
| Several documents of different usefulness | nDCG@10 with graded labels |
| Comparing two rerankers | Paired per-query nDCG with bootstrap CI, not two averages |
Cost maths
Labels dominate: 300 queries × 15 judged docs × 25 s ≈ 31 hours ≈ $1,900 at $60/h. Pooling the top 10 from 3 systems typically yields 15–20 unique docs per query to judge.
Operating it
- New retriever or reranker returns many unjudged docs in its top 10: re-pool and label
- Decisions on deltas < 0.02 without a significance test
- Per-slice nDCG spread > 0.15: slice-specific fixes
- nDCG@10 nightly per slice; alert on a > 0.03 drop
- Judged@10 coverage for each evaluated system; alert < 90%
- Per-query nDCG distribution; investigate queries that drop > 0.3 between releases
When it goes wrong
A new dense retriever scored 0.12 lower nDCG@10 than BM25 offline, but online click-through rose 9% in the A/B test
Offline and online results disagreed; a review found 38% of the new system's top 10 had never been judged
The golden labels were pooled only from BM25 results; unjudged relevant docs counted as 0, penalising the system that found different documents
Re-pooled labels from both systems, judged the new documents, and added 'judged@10' coverage to every report; the corrected offline delta was +0.04 for dense
Related terms
Sources
- Cumulated gain-based evaluation of IR techniques — Järvelin & Kekäläinen, 2002paper
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Thakur et al., 2021paper
- Introduction to Information Retrieval (ch. 8, Evaluation) — Manning, Raghavan & Schütze, 2008book
Check yourself
Three questions. Every option is explained once you answer.