MTEB leaderboard
Massive Text Embedding Benchmark: the public leaderboard for embedding models. Good for a shortlist; your own golden set decides.
MTEB is a big public scoreboard for Embedding models. It tests each model on many datasets and task types, such as search, grouping similar texts and classification, and ranks them.
It is a useful way to pick three or four models to try, but it measures public datasets, not your documents and your users' questions, so the final choice should come from a test on your own data.
A car magazine's road test: it narrows the list, but you still drive the two finalists on your own commute before buying.
Choosing the model with the best overall average. The mean blends tasks you do not run, gaps at the top are inside noise, and the winner may be 10× more expensive to serve for no gain on your data.
MTEB (Muennighoff et al., 2022) launched with 58 datasets across 8 task types: retrieval, reranking, STS, classification, clustering, pair classification, bitext mining and summarization. Retrieval tasks are BEIR-style and scored with nDCG@10. MMTEB (2025) expanded it to 500+ tasks in 250+ languages and introduced curated subsets, and the leaderboard now lets you filter by language, domain, model size and zero-shot status.
For RAG, read the retrieval column for your language, not the overall mean, which mixes in clustering and classification. Then filter on what the mean hides: parameters (0.6B vs 8B changes serving cost ~10×), Embedding dimensionality, max input tokens, licence, and whether the model trained on the benchmark's training splits. Top scores are often separated by less than one point, which is inside the noise of a 300-query domain eval. Treat the leaderboard as a shortlist of 3–5, and decide on your own Golden set with recall@k at the k your pipeline actually uses.
Shortlist 3–5 models from the MTEB retrieval tab filtered by language and size, always including one API model (voyage-3.5, text-embedding-3-large, gemini-embedding-001 or Cohere embed-v4) and one self-hostable model (bge-m3 or Qwen3-Embedding-0.6B). Decide on a 200–500-query golden set at your production k.
Financial filings RAG, English plus some German. Shortlist from the MTEB retrieval column, as of Oct 2026: Qwen3-Embedding-8B (top of multilingual, 4096 dims), Qwen3-Embedding-0.6B (1024 dims), bge-m3 (568M, 1024 dims), voyage-3.5 (API), text-embedding-3-large (API).
On a 400-query golden set of analyst questions, recall@20: Qwen3-8B 0.91, voyage-3.5 0.90, Qwen3-0.6B 0.88, bge-m3 0.87, 3-large 0.86. The 8B model needs an L40S-class GPU; voyage-3.5 is an API call at $0.06/M tokens. The team picked voyage-3.5 and kept bge-m3 as the self-hosted fallback.
Asked how you chose your embedding model, say: "MTEB retrieval in our language gave me five candidates; a 400-query golden set at recall@20 picked the winner, and cost per million chunks broke the tie." Expect "why not just take #1?" Because the top three are usually within a point, the mean includes tasks you don't run, and some top entries trained on the eval's sibling data.
Numbers worth knowing
- Original MTEB scope
- 58 datasets, 8 task types, 112 languagesMuennighoff et al., 2022
- MMTEB scope
- 500+ tasks, 250+ languages2025 release
- Typical gap between top-3 retrieval scores
- <1 nDCG@10 pointpublic leaderboard, as of Oct 2026
- Golden-set size to separate models 2 points apart
- ≥300 queriesrough bootstrap rule of thumb
How to decide
| When | Do this |
|---|---|
| Two candidates are within 1 point on your golden set | Pick the cheaper or self-hostable one; the difference is noise. |
| Your domain is far from the web (part numbers, clinical shorthand) | Expect leaderboard rank to mislead; budget for Fine-tuning an embedding model or Hybrid search. |
| Data residency forbids external APIs | Restrict the shortlist to open-weights models you can host. |
Cost maths
Eval cost is small; serving cost is not. 20M chunks × 400 tokens = 8B tokens. API: voyage-3.5 at $0.06/M = $480; text-embedding-3-large at $0.13/M = $1,040 (list prices, Oct 2026). Self-hosted Qwen3-Embedding-8B at ~1.5k passages/s on one L40S (estimate) = ~3.7 GPU-hours ≈ $7 at an assumed $1.90/h, plus the ops cost of owning a GPU service.
Operating it
- A new model tops the retrieval tab by >2 points in your language: re-run the golden set (quarterly)
- Query volume makes per-query embedding cost exceed ~$500/month: evaluate self-hosting
- Golden-set recall@k for the live model on every corpus refresh: alert on a 2-point drop
- Quarterly leaderboard check in your language/size bracket: ticket if a candidate leads by >2 points
When it goes wrong
After switching to the #1 overall MTEB model, a support assistant's answer quality dropped and latency doubled.
Online thumbs-down rate rose from 9% to 14%; the offline golden set had never been run on the new model.
The model led on classification and clustering; its retrieval score in the product's language was 4th, and it was 7B parameters vs the prior 335M.
Rolled back, made a golden-set run a required Eval gate (CI) for any embedding change, and selected on the retrieval column.
Sources
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al., 2022paper
- MMTEB: Massive Multilingual Text Embedding Benchmark — Enevoldsen et al., 2025paper
- Gemini Embedding: Generalizable Embeddings from Gemini — Lee et al., 2025paper
Check yourself
Three questions. Every option is explained once you answer.