Anatomy of a production RAG system
Every box between the user's question and a cited answer, with a latency and cost budget per stage, who owns it, and the one metric that tells you it is broken.
A demo RAG system is four boxes: embed, search, prompt, generate. A production one is closer to fifteen, split across two paths that run on different clocks. The ingestion path runs when documents change and is measured in minutes of freshness. The query path runs on every question and is measured in milliseconds to first token. Most outages happen at the seams between boxes owned by different people, so this post names an owner for each.
The worked example is an internal engineering wiki plus developer docs: 2M chunks, 50 queries per second at peak, answers streamed with citations. All numbers below are budgets under stated assumptions, not benchmarks. Re-derive them for your hardware and provider.
The query path, end to end
Walk the boxes in order:
- Gateway. Authentication, tenant resolution, rate limits, an exact-match or semantic cache, and the LLM gateway functions: provider keys, fallback, cost metering. Budget: 5–10 ms.
- Query rewriter. Turns “what about for staging?” into a standalone query using conversation history. A small model (Haiku-class or an 8B open-weights model), 200–400 input tokens, ≤60 output. Budget: 150–300 ms. Skip it for single-turn traffic; see query rewriting.
- Query embedding. One call to the same model that embedded the corpus. Budget: 15–40 ms hosted, under 10 ms on a co-located GPU.
- Retrieval. Hybrid search: BM25 in OpenSearch and ANN over an HNSW index, in parallel, both filtered by ACL and tenant metadata inside the query, then fused with RRF. Budget: 20–50 ms at p50 for 2M chunks; p95 is set by the slower of the two.
- Reranker. A cross-encoder such as bge-reranker-v2-m3 (568M) scores 50 candidates. Budget: 80–150 ms on one A10G-class GPU. This is the single biggest quality lever after chunking; see rerankers.
- Context assembly. Pick the top 4–8 chunks, dedupe near-duplicates, order the best ones first and last (models under-use the middle, per Lost in the Middle), attach stable citation ids, cap at a token budget. Budget: under 5 ms. See context assembly.
- Generation. The big model. Time to first token is 300–800 ms for a 4–6k token prompt on a hosted frontier model; prompt caching on the stable system prompt cuts that and the input bill. Then 300 output tokens stream at 50–100 tokens/s.
- Post-processing. Citation check (every cited id must be in the assembled context), PII and policy guardrails, and abstention when the reranker's best score is below threshold. Runs on the stream or right after it.
The latency budget
Set a target for time to first token, not for full answer time: users tolerate a long answer that starts quickly. Assume an SLO of TTFT ≤ 1.2 s at p50 and ≤ 2.5 s at p95 for this wiki assistant.
Two things fall out of the bars. First, the two model calls (rewrite and generation) own about 80% of the budget, so retrieval tuning will not rescue a slow TTFT; a smaller rewriter or skipping it for first-turn questions will. Second, p95 is dominated by provider variance, which is why the gateway needs a fallback with a timeout shorter than your p95 budget.
The cost budget
Assume list prices, illustrative as of Oct 2026: generator $3 / M input and $15 / M output tokens; rewriter $0.80 / M input and $4 / M output; embeddings $0.02 / M tokens; reranker on one A10G at $1.00/hour serving 25 queries/s; vector and BM25 cluster at $1,500/month for 50M queries/month. Prompt: 1,500 cached system tokens (reads at 0.1× base) + 4,000 retrieved tokens + 100 question tokens; 300 output tokens.
| Stage | Per query | Share | Owner |
|---|---|---|---|
| Generation input | $0.0128 | 68% | Applied AI |
| Generation output | $0.0045 | 24% | Applied AI |
| Rewriter | $0.0005 | 2% | Applied AI |
| Rerank GPU | $0.00001 | <1% | ML platform |
| Search cluster | $0.00003 | <1% | Search infra |
| Embedding | <$0.00001 | <1% | ML platform |
| Eval + judge sampling (5%) | $0.0009 | 5% | Eval owner |
| Total | ~$0.019 | 100% |
The arithmetic: input = 4,100 × $3/M + 1,500 × $0.30/M ≈ $0.0128; judge sampling = 5% × (~5,000 input + 200 output tokens at generator prices) ≈ $0.0009; output = 300 × $15/M = $0.0045. The lesson matches the latency one: every chunk you put in the prompt costs about $0.0015 at 500 tokens, every time. Cutting from 8 chunks to 5 after a good reranker saves more money than any infrastructure change in the table. Track it as cost per query, per tenant.
The ingestion path
Each box has one job: change data capture or webhooks from sources; layout-aware parsing so tables survive; chunking with headings attached (contextual enrichment); a content hash so unchanged chunks are never re-embedded; batch embedding (batch APIs are ~50% cheaper); writes to both indexes with ACLs copied from the source; and a document registry mapping each document to its chunk ids so deletes become tombstones everywhere, not orphans in one index.
Ownership and monitoring
| Box | Owner | Monitor | Alert when |
|---|---|---|---|
| Connectors + parse | Data platform | docs ingested/hour, parse errors | error rate > 1% or lag > 30 min |
| Chunk + embed | ML platform | embed queue depth, model version | queue > 1 h or version mismatch |
| Indexes | Search infra | p95 search latency, ANN recall probe | p95 > 80 ms or recall@10 < 0.95 |
| Rewrite + rerank | Applied AI | rerank top score distribution | median top score shifts > 15% |
| Generation | Applied AI | TTFT p95, tokens/query, fallback rate | TTFT p95 > 2.5 s or fallback > 2% |
| Answer quality | Eval owner | faithfulness, citation validity, abstain rate | faithfulness < 0.9 on daily sample |
| Access control | Security | cross-tenant leak canary | any hit |
Observability ties it together: one trace per query with a span per box, carrying the chunk ids retrieved, the reranker scores, the prompt version, and the model. Without the chunk ids in the trace you cannot answer the first question of every incident, which is “did retrieval or generation fail?” Sample 2–5% of traces into an LLM judge for faithfulness and citation validity, and keep a fixed golden set that gates every index, chunking or prompt change.
Key takeaways
- Production RAG is two systems: an ingestion path measured by freshness and a query path measured by time to first token.
- The two model calls own ~80% of latency and over 90% of cost; retrieval is cheap, so spend its budget on quality (hybrid + rerank).
- Retrieved context is the bill. Every chunk you can drop after a good reranker is roughly $0.0015 a query at the assumed prices.
- Name one owner per box and one alerting metric per box; put chunk ids and scores in every trace.
- Filter ACLs inside the query and prove it with a cross-tenant canary.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- Reciprocal rank fusion outperforms Condorcet and individual rank learning methods — Cormack et al., 2009paper
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov & Yashunin, 2016paper