Vector space
The high-dimensional space embeddings live in, where the angle or distance between vectors stands for difference in meaning.
Once texts are turned into embeddings, each one is a point in a space. Texts about similar things cluster together; unrelated texts sit far apart. That space is the vector space.
We draw it in 2D for intuition, but real ones have hundreds or thousands of dimensions. Search becomes 'find the points nearest my question's point.'
A supermarket floor plan: dairy items sit together, cleaning products sit together, and you find things by walking to the right neighbourhood.
Carrying a fixed similarity cutoff from one model to another; score distributions shift and the cutoff silently breaks.
Each embedding model defines its own space; geometry only means something within one model and version. Closeness is measured with Cosine similarity, Dot product or Euclidean distance; for L2-normalised vectors all three give the same ranking.
High dimensions are unintuitive. Random unit vectors in d dimensions are nearly orthogonal: their cosine has a standard deviation of about 1/√d, ≈0.03 at d = 1,024. Real embedding models are anisotropic (Ethayarajh, 2019): vectors crowd into a narrow cone, so unrelated texts can score 0.6–0.8 on some models and near 0.1 on others. Absolute thresholds therefore never transfer between models. Exact nearest-neighbour search costs N × d multiply-adds per query, which is why production uses Approximate nearest neighbour (ANN) indexes (HNSW, IVF (inverted file index), Product quantization (PQ)) that trade 1–5% ANN recall (efSearch, nprobe) for 100×+ speedups. 2D projections (UMAP, t-SNE) are good for spotting clusters and duplicates but distort distances; never tune thresholds from them.
One model per index, version recorded; L2-normalise and use inner product; exact search (FAISS flat or pgvector without an index) below ~1M vectors, HNSW above; thresholds calibrated per model on 300–1,000 labelled pairs; UMAP plots for exploration only.
E-commerce catalog, 768-dim space: hiking boots, trail runners and gaiters form one neighbourhood; kitchen kettles another. The query 'waterproof boots for muddy trails' lands in the footwear cluster; its 5 nearest products all have cosine between 0.71 and 0.79. Brute force over 2M products is 2M × 768 ≈ 1.5B multiply-adds per query; HNSW answers in ~2 ms at 0.97 recall.
Commit: similarity values are only meaningful within one model's space, so thresholds are calibrated per model on labelled pairs, and brute force is fine below ~1M vectors and replaced by HNSW above. Follow-up: 'we switched embedding models and the semantic cache started serving wrong answers?': the score distribution shifted; recalibrate the threshold on a few hundred labelled pairs before switching traffic.
Numbers worth knowing
- Random-vector cosine spread
- std ≈ 1/√d≈0.03 at d = 1,024; real models are anisotropic and sit higher
- Unrelated-pair cosine
- ≈0.1 to 0.8varies by model; measure the distribution before setting any threshold
- Brute-force cost
- N × d multiply-adds per query10M × 1,024 ≈ 10B and 41 GB streamed from memory: ~10 ms on a GPU, 0.2–0.4 s of a whole CPU socket
How to decide
| When | Do this |
|---|---|
| Fewer than ~1M vectors and latency budget ≥ 50 ms | Exact search; skip ANN complexity. |
| More than ~1M vectors or p95 under 20 ms needed | HNSW with recall measured against exact search. |
| Switching embedding models | Recalibrate every threshold on labelled pairs before cut-over. |
Cost maths
Exact search per query = N × d multiply-adds, and on CPU it is memory-bound: 10M × 1,024 float32 means streaming 41 GB per query, 0.2–0.4 s of a socket with 100–200 GB/s bandwidth, so 100 QPS would need 20–40 sockets. HNSW cuts that to ~1–5 ms per query on 1–2 cores but holds the graph plus vectors in RAM (~45–60 GB for 10M × 1,024 float32), so the cost moves from CPU to memory.
Operating it
- Exact search p95 above 50 ms
- ANN recall against exact search below 0.95 on a sampled query set
- Index no longer fits in one node's RAM
- Distribution of top-1 similarity per query (p10/p50/p90): alert on a shift above 0.05 after any model change
- ANN recall vs exact on 500 sampled queries weekly: alert below 0.95
When it goes wrong
After an embedding-model upgrade, a semantic cache on a product-help route served the wrong cached answer to ~6% of queries.
Spike in 'that's not what I asked' feedback, then an offline replay of 1,000 cache hits labelled by hand.
The 0.85 cosine threshold was tuned on the old model; the new model was more anisotropic and scored unrelated questions around 0.80–0.88.
Threshold recalibrated on 800 labelled pairs to 0.93 for 99% precision, and threshold calibration added to the model-upgrade checklist.
Related terms
Where it shows up
Sources
- How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings — Ethayarajh, 2019paper
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov & Yashunin, 2016paper
Check yourself
Three questions. Every option is explained once you answer.