How would you migrate to a new embedding model in production?
- 1 min — vector spaces are incompatible
- 3 min — parallel index, dual-write, backfill, shadow, ramp
- 2 min — cost and time estimate
- 1 min — thresholds and rollback window
Vector spaces are incompatible — you cannot mix old and new vectors in one index, ever. So: build a parallel index, backfill with the new model, shadow-read both to compare, then cut over behind a flag with the old index retained for rollback. Never re-embed in place.
Sequence it as: (1) freeze a golden eval set; (2) build the new index from the document registry, not from the old index — you need the original text; (3) run shadow traffic and diff the top-k; (4) ramp 1% → 10% → 50% → 100% watching quality and latency; (5) keep the old index for one rollback window, then delete.
Three things people miss: dual-write during backfill so documents changing mid-migration land in both indexes; cost and time estimation up front (embedding 50M chunks is a real budget line and a multi-day job); and the fact that any score thresholds you calibrated are now invalid and must be re-tuned on the new space. Org scope: the migration is a cross-team programme — the platform team owns the parallel index and dual-write, each consuming team signs off on its own golden set before its tenants ramp. Build the shadow-compare tooling once; you will swap models again within 18 months.
Assumptions: 40M chunks × 400 tokens = 16B tokens; 1024-dim float32 vectors.
- Hosted at $0.02 / M tokens (illustrative, Oct 2026): ~$320; at $0.13 / M: ~$2,100. At an assumed 5M tokens/min rate limit: 3,200 min ≈ 2.2 days.
- Self-hosted bge-m3 (568M): assume ~400 chunks/s per A10G at 400 tokens (measure — varies 2× with batch size) → 100k GPU-seconds ≈ 28 GPU-hours; 4 GPUs ≈ 7 h; at ~$1/GPU-hour ≈ $30.
- RAM: 40M × 4 KB = 164 GB per index; both indexes live during migration = ~330 GB plus HNSW links. Halving dims to 512 saves ~82 GB on the new side.
Money is small; calendar time, rate limits and double RAM are what you plan around.
What the answer sounds like at each level
Re-embed everything with the new model and switch over.
Parallel index, backfill from source text, shadow reads, flagged ramp, rollback window.
Adds dual-write during backfill, up-front cost and time estimates, threshold re-calibration, and per-tenant ramp with consuming-team sign-off.
Follow-ups the interviewer will push
1Now the corpus is 10× (400M chunks).
Backfill time and RAM dominate. Self-host the embedding model across a GPU pool, shard the new index, consider Matryoshka truncation or product quantization to keep memory flat, and plan for weeks, not days — start well before the deprecation date.
2Now the old provider shuts the model off mid-migration.
Queries can no longer be embedded for the old index. Cut reads to the new index for tenants already backfilled, serve BM25-only for the rest with a degraded banner, and prioritise backfill by traffic.
Red flags interviewers listen for
- Re-embeds in place in a live index.
- Builds the new index from old vectors instead of source text.
- Forgets that thresholds must be re-calibrated.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Safety | Parallel index, dual-write and rollback window. |
| Estimation | Sizes tokens, GPU hours and RAM before starting. |
| Org scope | Per-tenant ramp with consuming-team sign-off. |
Re-embedding a live index
Illustrative exampleOur embedding provider announced deprecation of the model backing a 40-million-chunk index serving several internal products, with a hard cutoff about four months out.
I led the migration across three consuming teams. The constraint was no quality regression and no read downtime, on a corpus that changed continuously.
I started with the eval, not the infra: we froze a 400-query golden set with labelled relevance and benchmarked four candidate models, including a smaller one at half the dimensions. The smaller model scored within noise of the largest on our domain, which cut index memory roughly 45%. For the cutover I built a parallel index and a dual-write path so live edits landed in both, backfilled over nine days with a rate-limited worker pool, then ran a week of shadow reads diffing top-10 overlap and reranker scores. We ramped behind a per-tenant flag. The step I had to argue for was re-calibrating the abstention thresholds on the new space — the old values would have silently doubled our refusal rate.
Cut over with no downtime and a small quality improvement on the golden set, at lower ongoing cost. I wrote the dual-write and shadow-compare pieces as a reusable migration module; the next model swap, a year later, took one engineer about two weeks instead of a quarter.
Sources
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al., 2022paper
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov & Yashunin, 2016paper