What happens internally when a RAG query is processed?
- 1 min — the stage list, end to end
- 3 min — walk one trace with per-stage latency
- 2 min — context assembly and why ordering matters
- 1 min — agentic loops and tail latency
Query → understanding (rewrite, expand, classify intent, extract filters) → embed and/or tokenize → candidate retrieval (ANN search + lexical search, with metadata filters applied) → fusion → rerank with a cross-encoder → context assembly (dedupe, order, budget tokens) → generation with a grounded prompt → post-processing (citation binding, groundedness check) → response.
Call out the two places people forget: context assembly (ordering and deduplication change answer quality more than most retrieval tuning) and the latency budget per stage. Also note that retrieval is often not a single pass — agentic RAG loops retrieval when the first pass is insufficient, at the cost of tail latency.
Org scope: each stage tends to get its own owner — search infra owns retrieval, ML owns the reranker, the product team owns prompts. Publish a per-stage latency budget as a shared contract and trace every request end to end (OpenTelemetry spans per stage), or regressions bounce between teams for weeks.
Assumptions: 1 query, 50 ANN + 50 BM25 candidates, rerank 50 with bge-reranker-v2-m3 (568M) on one A10G, generate 300 tokens.
- Rewrite (small model, cached): 10–50 ms
- Embed query: 5–15 ms
- ANN (HNSW, 10M vectors) + BM25 in parallel: 20–40 ms
- RRF fusion: <1 ms
- Rerank 50 pairs: 60–150 ms
- Assembly: <5 ms
- Generation: TTFT 300–600 ms, then ~30–60 ms/token → 1–2 s total
Total p50 ≈ 1.5–2.5 s, of which ~75% is generation. Optimise there first.
What the answer sounds like at each level
Lists the stages in order: embed the query, search, take top-k, put them in the prompt, generate.
Adds query understanding, hybrid retrieval with fusion, reranking and context assembly, and gives a per-stage latency budget.
Walks a real trace with numbers, says where the p99 lives (generation and agentic retries), and names which stage each failure class points to and who owns it.
Follow-ups the interviewer will push
1Now p95 must drop from 2.5 s to 1.2 s at the same quality.
Generation is ~70% of the budget, so stream tokens (TTFT, not total latency, is what users feel), shrink context from 8 to 4 chunks after the reranker, and route easy intents to a smaller model. Retrieval stages are already <150 ms; shaving them is rounding error.
2Now the reranker service is down.
Degrade, don't fail: fall back to the fused hybrid order with a smaller top-k, flag the response as degraded in logs, and alert. Quality drops a few points of nDCG@10, which beats a 5xx.
Red flags interviewers listen for
- Describes only embed → search → generate with no reranking or context assembly.
- Cannot attach even rough latencies to stages.
- Ignores metadata filters and ACLs during retrieval.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Completeness | Includes query understanding, hybrid retrieval, fusion, rerank, assembly and post-checks. |
| Quantitative | Gives a latency budget that sums and identifies generation as dominant. |
| Operability | Mentions per-stage tracing and degradation paths. |
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods — Cormack et al., 2009paper