Embedding dimensionality
How many numbers are in each embedding (384, 768, 1024, 1536, 3072). Memory, latency and cost scale linearly with it; quality does not.
An Embedding turns text into a list of numbers, and dimensionality is the length of that list. A 768-dimensional embedding stores 768 numbers for every piece of text.
Longer lists can capture finer shades of meaning, but every stored chunk carries all of them, so storage and search cost grow in direct proportion. Past a point the extra numbers buy very little, so the right size is the smallest one that still finds the right answers on your own test questions.
Describing a used car with 5 facts (make, year, mileage, price, colour) versus 500. The 500-fact sheet separates near-identical cars, but the filing cabinet is 100 times heavier and most buyers only ever check a dozen facts.
Choosing the largest dimension because a leaderboard model uses it. The RAM and latency cost is certain and linear; the quality gain past ~1024 dims is often under one point on domain data.
The output layer fixes d: 384 for all-MiniLM-L6-v2 (22M params), 768 for nomic-embed-text-v1.5, 1024 for bge-m3 (568M), e5-large-v2 and voyage-3.5 by default, 1536 for text-embedding-3-small and Cohere embed-v4 by default, 3072 for text-embedding-3-large and gemini-embedding-001 (as of Oct 2026). Raw storage is N × d × bytes; float32 is 4 bytes, int8 1, binary 1/8.
Cost is linear in d for RAM, distance arithmetic, network transfer and HNSW cache misses. Quality is sublinear: OpenAI reports text-embedding-3-large scoring 64.6 MTEB average at 3072 dims, 64.1 at 1024 and 62.0 at 256. The 12× cut from 3072 to 256 costs 2.6 points. Three levers cut d's cost without a model change: Matryoshka embeddings truncation (only on models trained for it), scalar or binary Scalar & binary quantization, and Product quantization (PQ). Any change to d is a new index and a full Backfill; vectors of different sizes or models cannot share an index. pgvector's HNSW index caps the vector type at 2,000 dims (use halfvec up to 4,000), so 3072-dim models need halfvec or truncation there.
Start at 1024 dims: bge-m3 self-hosted, or text-embedding-3-large / gemini-embedding-001 truncated to 1024, or voyage-3.5 at its 1024 default. Store int8 in the ANN index and full-precision on disk for rescoring. Move to 512 or 256 only after the golden set says recall holds.
Insurance claims archive, 20M chunks. text-embedding-3-large at full 3072 dims in float32 is 246 GB before index overhead. Truncated to 1024 dims it is 82 GB; at 1024 dims int8, 20 GB; at 1024 dims binary with int8 rescoring from disk, 2.6 GB in RAM.
On a 500-query golden set of adjuster questions, recall@10 went 0.912 (3072 fp32) → 0.905 (1024 fp32) → 0.899 (1024 int8). The team shipped 1024 int8: 12× less RAM for 1.3 points of recall.
For a capacity question, do the arithmetic out loud: "50M chunks at 1536 float32 is 307 GB of raw vectors plus ~30% for the HNSW graph. I'd test 768 or 1024 via Matryoshka, then int8, and accept anything within 1 point of full recall on our golden set." The follow-up is usually "what if recall drops 3 points?" Answer: keep the small first-pass index and rescore the top 100–200 with full-precision vectors from disk.
Numbers worth knowing
- Raw float32 storage
- 4 KB per 1024-dim vector; 4.1 GB per 1Mbefore index overhead
- text-embedding-3-large MTEB avg
- 64.6 @3072 · 64.1 @1024 · 62.0 @256OpenAI-reported, Jan 2024
- HNSW graph overhead
- ~100–200 B per vectorM=16, independent of d
- pgvector HNSW index limit
- 2,000 dims (vector), 4,000 (halfvec)pgvector 0.7+
How to decide
| When | Do this |
|---|---|
| The model supports Matryoshka (text-embedding-3, gemini-embedding-001, voyage-3.5, nomic-embed v1.5, Qwen3-Embedding) | Truncate to 1024 or 512 and re-normalize; no retraining needed. |
| The model does not support Matryoshka and RAM is the bill | Keep native d and quantize to int8 (≈4× smaller, ~97–99% of retrieval quality). |
| Corpus is under 1M chunks | Don't optimize d at all: 1M × 3072 fp32 is 12 GB, which fits on one node. |
Cost maths
RAM = N × d × bytes × 1.3 (graph + headroom). 50M chunks: 3072 fp32 = 800 GB; 1024 int8 = 67 GB. Assume ~$5 per GB-month of RAM (illustrative, memory-optimized cloud instances, Oct 2026): $4,000/month vs $335/month. Re-embedding cost is separate and one-off: 50M chunks × 400 tokens = 20B tokens × $0.13/M (text-embedding-3-large list price, Oct 2026) = $2,600, or $1,300 via the Batch API.
Operating it
- Vector RAM above ~60% of node memory: truncate or quantize before sharding
- HNSW p95 latency rising with corpus size while CPU is idle: memory-bound, reduce bytes per vector
- Corpus crossing 10M chunks at ≥1536 fp32: cost of d now exceeds the cost of a re-embed
- Index RAM / node RAM: alert above 75%
- Golden-set recall@10 after any d or precision change: block rollout on a drop >1 point
- Page faults per second on search nodes: alert on sustained >1k/s (index spilling to disk)
When it goes wrong
Financial-filings search p95 jumped from 40 ms to 900 ms after a model upgrade to a 3072-dim model.
Latency SLO alert plus node memory at 97% with heavy page-cache eviction.
The index was rebuilt at 3072 fp32 on nodes sized for 1024 dims; vectors no longer fit in RAM and HNSW traversal hit disk.
Rebuilt at 1024-dim Matryoshka truncation with int8, re-ran the golden set (−0.6 recall@10), and added a pre-backfill capacity check.
Related terms
Sources
- Matryoshka Representation Learning — Kusupati et al., 2022paper
- Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval — Shakir, Aarsen & Lee (Hugging Face), 2024blog
- Embeddings guide — OpenAI, 2024docs
- pgvector: open-source vector similarity search for Postgres — pgvector, 2024docs
Check yourself
Three questions. Every option is explained once you answer.