Matryoshka embeddings
Embeddings trained so the first N dimensions work alone: cut 3072 to 256 and keep ~96% of quality. Re-normalize after cutting.
Matryoshka embeddings are trained so the front of the vector carries the most important information. You can keep only the first part, for example 256 of 1024 numbers, and the shorter vector still works well.
That lets you trade a small amount of quality for much smaller storage and faster search without changing models. It only works with models that were trained this way, and you must rescale the shortened vector to length 1 before comparing.
Russian nesting dolls: every smaller doll is a complete doll, just with less detail painted on.
Truncating a model that was not trained with the nested loss and expecting graceful degradation, or truncating without re-normalizing, which silently breaks inner-product ranking.
Matryoshka Representation Learning (Kusupati et al., 2022) sums the training loss over nested prefixes, for example 64, 128, 256, 512, 1024 and full, so every prefix is a usable embedding. Supported as of Oct 2026: text-embedding-3-small/large (dimensions parameter), gemini-embedding-001 (output_dimensionality, recommended 768/1536/3072), voyage-3.5 and voyage-3-large (256/512/1024/2048), Cohere embed-v4 (256/512/1024/1536), nomic-embed-text-v1.5 (64–768), Qwen3-Embedding. Plain BERT-style models without the nested loss degrade sharply when truncated.
After truncating, L2-normalize again; the prefix of a unit vector is not a unit vector. The production pattern is funnel retrieval: an HNSW index over a 256-dim (or binary) prefix in RAM returns 100–500 candidates, then the full-precision vectors, kept on disk or in a cheaper store, rescore them. This recovers most of the full-dimension quality at a fraction of the RAM. Truncation is applied per index, so you can run 256 dims for a high-QPS autocomplete and 1024 for the main RAG index from one embedding call.
text-embedding-3-large or gemini-embedding-001 at 1024 dims for the main index; 256-dim prefix plus rescoring when RAM dominates. Always re-normalize after truncation. Store the full vector once (disk or object store) so you can change the prefix later without paying for another embedding call.
E-commerce catalog, 30M product chunks, text-embedding-3-large.
- Full 3072 fp32: 369 GB RAM.
- First pass on 256 dims fp32: 31 GB, top-200 candidates in ~6 ms.
- Rescore the 200 with 1024-dim int8 vectors from local NVMe: 200 KB read, ~2 ms.
Golden-set recall@10: 0.874 full 3072 vs 0.869 funnel. RAM down 12×, p95 down from 35 ms to 11 ms.
When vector RAM is the biggest bill, say: "If the model is Matryoshka-trained, I index a 256- or 512-dim prefix, oversample 4–10×, and rescore with full vectors; otherwise I quantize instead, because truncating a non-MRL model loses far more." Expect "how do you choose the prefix?" Sweep 128 to full on the golden set, plot recall against RAM, and pick the knee where one more doubling buys less than one point.
Numbers worth knowing
- Quality at 256 of 3072 dims
- 62.0 vs 64.6 MTEB (≈96%)text-embedding-3-large, OpenAI-reported
- RAM saving at 256 dims
- 12× vs 3072, 4× vs 1024same precision
- Oversampling for rescoring
- 4–10× final ke.g. top-200 to keep top-20
How to decide
| When | Do this |
|---|---|
| The model is MRL-trained and RAM is the dominant cost | Index a 256–512-dim prefix, rescore top 4–10× k with the full vector. |
| The model is not MRL-trained | Do not truncate; use int8 or binary Scalar & binary quantization instead. |
| Recall at the short prefix is within 1 point of full | Skip rescoring and drop the full vectors from RAM entirely. |
Cost maths
RAM = N × d × bytes. 30M chunks: 3072 fp32 = 369 GB ≈ $1,850/month at an assumed ~$5/GB-month (illustrative, Oct 2026); 256 fp32 = 31 GB ≈ $155/month; full 1024 int8 on NVMe ≈ 31 GB at ~$0.10/GB-month ≈ $3. Embedding cost does not change: you pay for one 3072-dim call and truncate locally.
Operating it
- Index RAM above 60% of node memory and recall headroom ≥1 point: halve the prefix
- Rescore stage p95 above 10 ms: move full vectors from object store to local NVMe or RAM
- Recall@10 of funnel vs full-dimension baseline on the golden set: alert if gap >1.5 points
- Rescore-stage latency p95: alert above 10 ms
- Vector norm check at write: alert on any norm outside 1.000 ± 0.001
When it goes wrong
After switching to 512-dim truncation, catalog search ranked near-random for about 8% of queries.
Golden-set recall@10 fell 11 points; investigation found vector norms between 0.62 and 0.81.
Vectors were truncated client-side without re-normalization, in an inner-product index.
Requested truncation via the API dimensions parameter (which returns normalized vectors), added a norm assertion at write time, rebuilt the index.
Related terms
Where it shows up
Sources
- Matryoshka Representation Learning — Kusupati et al., 2022paper
- Embeddings guide — OpenAI, 2024docs
- Nomic Embed: Training a Reproducible Long Context Text Embedder — Nussbaum et al., 2024paper
- Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval — Shakir, Aarsen & Lee (Hugging Face), 2024blog
Check yourself
Three questions. Every option is explained once you answer.