Scalar & binary quantization
Storing embedding values in fewer bits — float16, int8, or 1-bit binary — to cut vector memory 2–32× with a rescoring pass to recover recall.
Each number in an Embedding is normally stored as a 32-bit float. Vector quantization stores it with fewer bits: 16-bit floats (half the memory), 8-bit integers (a quarter) or even a single bit per number (1/32). Search runs on the small version to find candidates, then the best ones are re-checked with the full-precision vectors. This is about stored vectors; shrinking an LLM's weights is Model quantization.
Saving photos as small thumbnails to browse quickly, then opening the full-resolution file only for the few you actually want.
Quantizing without keeping originals. Rescoring needs full-precision vectors somewhere (disk is fine); if you discarded them, recall loss is permanent and a re-embed of the whole corpus is the only fix.
float16/bfloat16: 2×, recall loss ≈ 0; pgvector halfvec, supported everywhere. Scalar int8 (SQ8): per-dimension min/max (or quantile) mapped to 0–255; 4×, typically ≥ 99% of float32 recall after light rescoring; Qdrant scalar quantization, FAISS SQ8, Elasticsearch int8_hnsw. Binary: sign bit per dimension, Hamming distance via popcount; 32×, 15–40× faster scans, recall depends heavily on the model — Hugging Face measured ~96% of float32 retrieval quality retained for some models after rescoring, much less for others; best with ≥ 1,024 dims. Elasticsearch BBQ and Qdrant binary/1.5-bit/2-bit modes add correction terms and oversampling. Combine with Matryoshka embeddings truncation (1024 → 256 dims) for another 4×. Rescoring pattern: fetch k × oversampling (2–4× for int8, 4–10× for binary) on the compressed index, reorder with float32 vectors kept on disk.
float16 storage by default (pgvector halfvec). int8 scalar quantization with originals on disk and 2× oversampling when RAM > 50% used (Qdrant scalar, Elasticsearch int8_hnsw, FAISS SQ8). Binary only with a ≥ 1,024-d model, 4–10× oversampling, and an offline recall check.
Customer-support corpus, 25M chunks × 1024-d (bge-m3). float32: 102 GB. int8 in Qdrant with always_ram: true, originals on disk, oversampling 2.0: 26 GB RAM, recall@10 0.985 of float32. Binary + oversampling 8 + rescore: 3.2 GB RAM, recall@10 0.94; p50 latency 4.1 ms → 1.3 ms. Team ships int8 — binary's 4% loss fails their 0.97 floor.
Recommend by tier: "float16 always, int8 when RAM is the constraint, binary only after measuring the model on my data with 4–10× oversampling and rescoring." Expect: why is binary model-dependent? Because sign bits keep only direction per dimension; models trained with quantization-aware or Matryoshka objectives keep more signal in signs.
Numbers worth knowing
- Memory reduction
- 2× / 4× / 32×float16 / int8 / binary vs float32
- Recall retained
- ≈100% / ≥99% / ~90–96%after rescoring; binary is model-dependent (HF 2024)
- Binary scan speedup
- 15–40×Hamming via popcount vs float32 dot products
How to decide
| When | Do this |
|---|---|
| Any vector store | Store float16; it is free recall-wise. |
| Index RAM > 50% of node | int8 with originals on disk and 2× oversampling. |
| Billions of vectors or latency-critical first stage | Binary (+ Matryoshka truncation) as a first pass, float rescoring of top 100–400. |
Cost maths
25M × 1024-d: float32 102 GB RAM → 2 × 64 GB nodes per replica (~$620/month at ~$0.43/hr, illustrative Oct 2026). int8: 26 GB → one 32 GB node (~$155/month) + 102 GB disk (~$8/month). With 2 replicas the int8 move saves ~$930/month. Binary saves another ~$120 but costs ~4 recall points — rarely worth it below 100M vectors.
Operating it
- Vector RAM > 50% of node — move float32 → int8
- > 100M vectors and int8 still needs > 4 nodes — evaluate binary/BBQ or Product quantization (PQ)
- Recall after quantization < floor — raise oversampling before abandoning the scheme
- Recall@10 vs float32 baseline on sampled queries — alert below 0.97
- Oversampling factor actually applied vs configured — alert on mismatch
- RAM headroom after quantization — alert below 25%
When it goes wrong
Code-search relevance dropped noticeably the day after enabling binary quantization; latency dashboards looked great.
Nightly golden-set recall@10 fell from 0.93 to 0.71.
The 384-d embedding model kept little information in sign bits, and oversampling was left at the default; no rescoring with originals.
Rolled back to int8 via config flag, documented that binary requires ≥ 1,024 dims and an offline recall check, and added recall@10 as a blocking metric for index-config changes.
Related terms
Sources
- Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval — Hugging Face, 2024blog
- Qdrant documentation: Quantization — Qdrant, 2026docs
- Matryoshka Representation Learning — Kusupati et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.