Embedding
A list of numbers (a vector) that represents the meaning of a text, so that similar meanings end up close together.
An embedding turns a piece of text into a fixed-length list of numbers, for example 1,024 numbers. The model that makes it is trained so texts with similar meaning get similar lists, even when they use different words.
That is what lets a search for 'check if the token expired' find a function called verify_jwt_expiry: their embeddings are close even though they share almost no words.
GPS coordinates for meaning: two places close on the map are close in meaning, whatever they are called.
Mixing vectors from two embedding models, or two versions, in one index. Their similarities are meaningless and recall drops with no error raised.
An Encoder pools token representations into one vector of 384–4,096 dimensions. Training is contrastive: pull query and relevant-passage pairs together, push in-batch and hard negatives apart (DPR). The result is a Vector space where Cosine similarity or Dot product approximates relevance.
Operational facts: vectors from different models, or different versions of one model, are not comparable, so a model change means a full re-embed and Backfill behind an index alias. Many models are asymmetric and need an Instruction prefix (query: / passage:) on queries. Matryoshka embeddings-trained models let you truncate dimensions (3,072 → 1,024) for 3× less storage at a small recall cost; int8 Scalar & binary quantization gives 4×, binary 32×. One vector per chunk compresses meaning lossily, so exact identifiers (error codes, SKUs, function names) are better matched by BM25; that is why Hybrid search is the default. Shortlist models on MTEB leaderboard, decide on your own Golden set.
bge-m3 (568M params, 1,024 dims, 8,192-token input) self-hosted, or a hosted embedding model truncated to 1,024 dims; L2-normalise at write; store model@version with every vector; hybrid with BM25; re-embed via a new index behind an Index alias.
Code search: the query 'check if the token expired' embeds to a 1,024-dim vector. Cosine with verify_jwt_expiry() is 0.78, with refresh_session_token() 0.71, with render_invoice_pdf() 0.12. Storing 10M code chunks as float32 takes 10M × 1,024 × 4 bytes = 41 GB; int8 takes 10 GB; binary 1.3 GB, usually with a float rescoring pass on the top 100.
Commit: the embedding model is a schema decision. Version it per vector, re-embed through a blue-green index swap, and run hybrid search because dense misses exact identifiers. Follow-up: 'storage is too expensive at 500M vectors?': Matryoshka truncation plus int8 or binary quantization with float rescoring, measured on recall@100.
Numbers worth knowing
- Dimensions
- 384–4,096common production models; 768–1,024 is the default sweet spot
- Float32 storage
- 4 bytes × dims per vector10M × 1,024 dims = 41 GB before index overhead
- Quantization savings
- 4× (int8), 32× (binary)typically with float rescoring on the top 100 to recover recall
How to decide
| When | Do this |
|---|---|
| Queries include identifiers, codes or exact names | Hybrid: dense plus BM25, fused with RRF. |
| Domain jargon dominates (legal, clinical, internal code names) | Fine-tuning an embedding model on 5k–50k in-domain pairs. |
| Vector storage is the main cost above ~100M vectors | Matryoshka truncation and int8/binary quantization, verified on recall@100. |
Cost maths
Hosted embedding cost = chunks × tokens per chunk × price. Assume $0.02 / M (small model) and $0.13 / M (large model), illustrative list prices, Oct 2026. 10M chunks × 400 tokens = 4B tokens → $80 or $520 per full embed; batch APIs halve it. Self-hosted bge-m3 on one L4 at an assumed ~$0.80/hour and ~300 chunks/s ≈ 9 hours ≈ $8. Storage dominates over time: 41 GB in RAM-resident HNSW is the line item to optimise.
Operating it
- Vector count above ~100M or index RAM above budget
- Recall@50 on the golden set plateaus below 0.85 after chunking fixes
- A new embedding model beats the current one by 3+ points on your golden set
- Recall@10 and @50 nightly on the golden set: alert at −3 points
- Distinct
model@versionvalues in the live index: alert if more than one - Embedding latency p95 at ingest: alert on 2× baseline
When it goes wrong
Code-search recall@10 fell from 0.84 to 0.61 over a week, with no deploy to the search service.
Nightly golden-set run.
A re-embedding job with a new model version was writing into the live index in place; half the vectors were from each model.
Blue-green re-embedding: build a new index, run the golden set on it, swap the alias atomically. Model version stored per vector and asserted at query time.
Related terms
Where it shows up
Sources
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020paper
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al., 2022paper
- Matryoshka Representation Learning — Kusupati et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.