How would you design a RAG system supporting millions of documents and high query traffic?
- 5 min — requirements: corpus size, QPS, latency SLO, recall target, cost ceiling.
- 10 min — architecture: ingestion plane, serving plane, sharded index, funnel, gateway.
- 10 min — capacity and cost estimate per tier.
- 5 min — degradation order, failure modes, re-indexing.
- 5 min — ownership, cost attribution and rollout.
Separate the planes and scale them independently. Ingestion is throughput-bound, bursty, and tolerant of latency — queue-driven autoscaling workers. Serving is latency-bound and must be predictable — horizontally scaled stateless services in front of a sharded index. Scale the index by partitioning (tenant, time, or topic) with a routing layer, replicate partitions for read throughput and availability, and keep the funnel so expensive stages only see small candidate sets.
Concretely, at this scale: memory is the binding constraint on HNSW, so quantise (binary quantisation with float rescoring over the top few hundred is a very good point on the curve); keep the hot partition in memory and cold data on disk-backed IVF; batch embedding calls; make the LLM gateway handle backpressure with queueing and graceful degradation rather than cascading timeouts.
- Define the SLO first. "Millions of documents" is not a requirement. 50ms p99 retrieval at 2,000 QPS with 95% ANN recall is. Make them give you numbers, or supply your own and state them.
- Degrade gracefully. Under load, shed the reranker before shedding results; return slightly worse answers rather than errors.
- Cost is a design constraint. At high QPS, embedding and LLM spend dominate infrastructure cost. Know your cost per thousand queries and design against it.
- Where it breaks: re-indexing. At 10M chunks × ~500 tokens ≈ 5B tokens, a full re-embed is hours of compute on a few GPUs (~$100 via a hosted embedding API at $0.02/M tokens), but days if you're rate-limited by the embedding API — plus the HNSW rebuild and 2× index capacity during cutover. Design the migration path on day one, not when the model deprecates.
- Org scope. Buy the vector store and LLM gateway, build the routing, funnel and cost attribution. The platform team owns the SLO and capacity plan; product teams get per-team cost-per-1k-query dashboards and quotas.
Assumptions (illustrative prices, Oct 2026): 2M documents → 10M chunks × ~500 tokens; 1024-dim embeddings; 2,000 QPS peak, 400 QPS average; 50ms p99 retrieval; end-to-end p95 2.5s; LLM at $3/M input, $15/M output; hosted embeddings at $0.02/M tokens.
- Re-index: 10M × 500 = 5B tokens → ~$100 via API. Self-hosted bge-m3 (568M) at ~75k tokens/s per H100 ≈ 18.5 GPU-hours → ~5 hours on 4 GPUs. At a 1M tokens/minute API rate limit: 5,000 minutes ≈ 3.5 days.
- Index memory: float32 41 GB; binary 1.3 GB + HNSW links 1.3 GB in RAM, float32 on NVMe for rescoring.
- Retrieval CPU: 2ms per query per shard × 4 shards × 2,000 QPS = 16 cores busy → 4 shards × 3 replicas on 8-vCPU nodes = 12 nodes × ~$0.50/h × 730 ≈ $4.4k/month.
- Reranker: 2,000 QPS × 50 candidates = 100k pairs/s ÷ ~3k pairs/s per A10G (MiniLM-class) ≈ 33 GPUs ≈ $24k/month at ~$1/GPU-hour — the first thing to shed under load.
- Concurrency: Little's law, 2,000 QPS × 2.5s = 5,000 requests in flight at the LLM gateway.
- LLM: 4,000 input + 300 output tokens = $0.012 + $0.0045 = $0.0165/query → $16.50 per 1k queries vs ~$0.004 per 1k for retrieval nodes. At 400 QPS average (~1B queries/month) that's ~$17M/month, so caching, routing easy queries to a ~10× cheaper model, and per-team quotas matter more than any index choice.
What the answer sounds like at each level
Use a scalable vector database, add more servers as traffic grows, and cache common queries.
Separates ingestion from serving, shards and replicates the index, quantises vectors, keeps the multi-stage funnel and sizes each tier against a stated SLO.
Starts from SLO and cost-per-1k-query targets, shows the LLM dominates spend by orders of magnitude, plans re-index capacity on day one and defines degradation order and team ownership.
Follow-ups the interviewer will push
1Traffic goes from 2,000 to 20,000 QPS. What scales linearly and what doesn't?
Retrieval nodes scale by adding replicas (linear). Reranker GPUs scale linearly and become expensive fast — cut candidates or shed it. LLM spend scales linearly and dominates; the levers are caching, smaller models for easy queries via routing, and provider rate limits, which may need reserved capacity or a second provider.
2The primary LLM provider is down. What does the user see?
The gateway's circuit breaker trips within seconds and routes to a fallback model (second provider or self-hosted) with its own pre-tested prompt; if that's saturated, return the retrieved passages with citations and no generated answer. Never cascade timeouts back into the retrieval tier.
3The embedding model is deprecated with 90 days' notice. Plan?
Blue-green: stand up a parallel index, re-embed 5B tokens over ~1 week at a rate that leaves headroom for live traffic, dual-write new documents, shadow-compare on golden sets per tenant, flip the alias. Budget 2× index memory for the overlap window.
Red flags interviewers listen for
- No SLO or QPS numbers before designing.
- Ignores LLM cost, which dominates at high QPS.
- No plan for re-indexing or model migration.
- Timeouts cascade instead of degrading gracefully.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Requirements | Turns 'millions of documents, high traffic' into explicit SLOs and targets. |
| Architecture | Separate planes, sharded and replicated index, multi-stage funnel. |
| Estimation | Memory, nodes, GPUs and $ per tier with stated assumptions. |
| Resilience | Explicit degradation order, provider failover and re-index plan. |
Sources
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov & Yashunin, 2016paper
- Product Quantization for Nearest Neighbor Search — Jégou et al., 2011paper
- Matryoshka Representation Learning — Kusupati et al., 2022paper
- Site Reliability Engineering, ch. 22: Addressing Cascading Failures — Google, 2016book