How do different embedding models affect retrieval quality?
- 1 min — axes of difference
- 2 min — domain eval as the decision
- 1 min — operational cost and vendor risk
- 1 min — Matryoshka and fine-tuning options
They differ in training domain, dimensionality, max sequence length, symmetric vs asymmetric design, and multilingual coverage. The biggest practical factor is domain match: a general-purpose model trained on web text handles medical or legal jargon poorly because the distinguishing terms were rare in training.
Benchmark leaderboards are a starting filter, not a decision. Build a domain-specific eval set — 200 real queries with labelled relevant documents — and run candidates against it. The leaderboard winner loses on specialised corpora often enough that you should always check.
Factor in the operational cost, not just the score: dimensionality drives index size and memory, sequence length constrains chunking, and a hosted model is a dependency with rate limits and a deprecation schedule. Mention Matryoshka embeddings as a way to trade dimensions for accuracy at query time. And note that fine-tuning an embedding model on in-domain pairs usually beats switching to a bigger general one. Org scope: model selection is a platform decision with a quarterly re-benchmark on each major tenant's golden set; teams should not pick their own embedding model, or you end up running five incompatible indexes.
What the answer sounds like at each level
Pick a model high on the MTEB leaderboard.
Builds a domain eval of ~200 labelled queries and weighs dimensionality, sequence length and multilinguality.
Adds operational cost, vendor deprecation risk and Matryoshka truncation, and often prefers fine-tuning a mid-size model over switching to a larger one.
Follow-ups the interviewer will push
1Now you need 30 languages.
Shortlist multilingual models (e.g. bge-m3, multilingual-e5-large, Cohere embed-multilingual) and build per-language slices of the golden set — aggregate scores hide a weak language.
2Now the hosted embedding provider has an outage.
Queries can't be embedded against an index built by that model, so failover to another model doesn't work. Fall back to BM25-only retrieval, or self-host an open-weights copy of the same model for continuity.
Red flags interviewers listen for
- Picks by leaderboard alone.
- Ignores index size and memory implications of dimensions.
- Assumes a hosted model has no deprecation schedule.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Evaluation | Domain-specific labelled set beats leaderboards. |
| Operations | Accounts for dimensions, sequence length, rate limits and deprecation. |
| Options | Considers truncation and embedding fine-tuning. |
Sources
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al., 2022paper
- Matryoshka Representation Learning — Kusupati et al., 2022paper