What are the main components of a production RAG system?
- 1 min — two planes: ingestion and serving
- 3 min — components in each, with the registry called out
- 2 min — eval harness and backfill as the improvement loop
- 1 min — build vs buy split
Two planes. Ingestion: connectors, parsers, chunker, enrichment/metadata extraction, embedding workers, vector + lexical index, a document registry with versions and ACLs. Serving: query understanding, retriever, reranker, context builder, LLM gateway, guardrails, cache. Across both: evaluation harness, observability, feedback capture, and a replay/backfill mechanism.
The component teams always under-build is the document registry — the source-of-truth table mapping source doc → version → chunks → vectors → ACL. Without it you cannot do deletions, re-embeds, or permission changes correctly.
The eval harness and the ingestion backfill pipeline are the two components that decide whether the system can be improved after launch. Everything else can be swapped. Say that out loud — it signals you've maintained a RAG system rather than built one. Org scope: build the document registry, eval harness and backfill tooling as one platform owned by a single team; buy the vector store and the LLM gateway. Product teams plug in connectors and prompts, never their own index.
What the answer sounds like at each level
Names a vector database, an embedding model, a retriever and an LLM.
Splits ingestion and serving planes, adds reranker, guardrails, cache and observability, and explains the document registry.
Says which components decide long-term quality (eval harness, backfill path), which are swappable commodities, and how ownership splits between platform and product teams.
Follow-ups the interviewer will push
1Now there are 30 product teams wanting RAG.
Offer a multi-tenant platform: shared ingestion and registry, per-tenant namespaces with ACL filters, a common eval harness each team feeds golden sets into, and a quota per tenant to stop noisy neighbours. Teams own prompts and connectors only.
2Now a regulated tenant needs data residency in the EU.
Registry and index become region-scoped: a separate EU deployment of the vector store and embedding endpoint, routing by tenant, and a deletion path that proves erasure within an SLA (e.g. 30 days, usually much faster).
Red flags interviewers listen for
- Equates the system with 'a vector DB plus an LLM'.
- No answer for deletion, permission changes or re-embedding.
- Omits evaluation entirely.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Architecture | Clear ingestion vs serving split with cross-cutting eval and observability. |
| Data lifecycle | Explains how deletes, ACL changes and re-embeds flow through the registry. |
| Org scope | Names platform vs product ownership and what is bought. |
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper