Estimate the time and cost to embed a 50M-chunk corpus, via an API and self-hosted.
- 1 min — chunks → tokens
- 3 min — API cost and time (rate limit), self-hosted GPU-hours
- 2 min — vector storage and compression options
- 2 min — migration mechanics: blue-green, idempotency, alias cutover
Convert chunks to tokens first (chunks × average tokens), then divide by throughput — API rate limit in tokens/minute, or measured GPU tokens/s for a self-hosted encoder. Cost is tokens × price for the API, GPU-hours × hourly rate for self-hosting. Then size the output: vectors × dimensions × bytes.
A developer-docs and code-search platform migrating 50M chunks (average 400 tokens) to a new embedding model. You need the backfill window, the bill, and whether the vector store has room for the output before the cutover plan is approved.
Embedding compute is the cheap part; the backfill is an ingestion operation. The risks are re-parsing (often slower than embedding), rate limits, and a mixed index if the cutover isn't atomic. Run it as a blue-green build into a new index behind an alias, idempotent per chunk keyed on content hash plus model version, and budget the next migration now — every model change re-pays this cost. The platform team owns the embedding service and model version; consuming teams never call the encoder directly.
Assumptions (illustrative, Oct 2026): 50M chunks × 400 tokens = 20B tokens. API: OpenAI text-embedding-3-small at $0.02 / M tokens, -large at $0.13 / M, batch API ~50% off; throughput limited to 5M tokens/minute on your tier. Self-hosted: bge-m3 (568M params, 1,024-d) on L4/A10G-class GPUs at ~$1/GPU-hour, measured ~12k tokens/s per GPU at 512-token sequences (benchmark yours — a 110M model runs ~5× faster).
- API cost: small = 20,000 × $0.02 = $400 ($200 batch); large = $2,600 ($1,300 batch).
- API time: 20B ÷ 5M/min = 4,000 min ≈ 67 hours — the rate limit, not price, is the constraint.
- Self-hosted: 20B ÷ 12k tok/s ≈ 1.67M GPU-seconds ≈ 460 GPU-hours ≈ $460; on 16 GPUs ≈ 29 hours.
- Storage: 50M × 1,024 × 4 B = 205 GB float32; 102 GB float16; 51 GB int8; 26 GB with Matryoshka truncation to 256-d float16.
- Hidden costs: re-parsing PDFs/HTML at ~20 docs/s per CPU core can exceed embedding time; dual-running two indexes during cutover doubles storage for the migration window.
What the answer sounds like at each level
Computes 20B tokens and the API bill from a list price.
Compares API vs self-hosted including rate limits and GPU-hours, and sizes the vector storage.
Frames it as a migration (blue-green, idempotent, alias cutover), notes the recurring cost of every future model change, and uses Matryoshka truncation or quantisation to control storage.
Follow-ups the interviewer will push
1Now the corpus grows 10× to 500M chunks.
API cost scales to $4k–26k and the rate limit becomes days, so self-hosting a smaller encoder (or Matryoshka 256-d output) on a spot GPU pool wins; storage at 1,024-d float32 is 2 TB, so binary or PQ compression becomes mandatory.
2Now 2% of chunks change daily.
That's 1M chunks/day ≈ 400M tokens ≈ $8/day on the small API model — incremental embedding via change data capture keyed by content hash is trivial; the full re-embed only happens on model change.
3Why not just keep the old model and skip the migration?
Only if the new model doesn't move recall on your eval set. Measure first on a 10k-query sample; if recall@10 gains are under ~2 points, the migration risk and dual-index window aren't worth it.
Red flags interviewers listen for
- Ignores the API rate limit and promises an overnight backfill.
- Forgets that query vectors and document vectors must come from the same model version — mixed indexes silently degrade.
- Doesn't size the output storage.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Arithmetic | 20B tokens, $400–2,600 API, ~460 GPU-hours, ~200 GB storage — units shown. |
| Constraint | Identifies throughput limits (rate limit or GPU count) as the real schedule driver. |
| Migration | Blue-green build, content-hash idempotency, model version in the key. |
| Forward-looking | Budgets the next model change and storage compression. |
Sources
- Embeddings guide — OpenAI, 2026docs
- BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation — Chen et al., 2024paper
- Matryoshka Representation Learning — Kusupati et al., 2022paper
- MTEB: Massive Text Embedding Benchmark — Muennighoff et al., 2022paper