Retrieval-Augmented Generation (RAG)
Retrieval-augmented generation: fetch relevant documents for a question, put them in the prompt, and have the LLM answer from them with citations.
RAG is a pattern for making an LLM answer from your own, current data. When a question arrives, the system first searches a document collection (the Corpus) for the most relevant passages, then pastes those passages into the Prompt & system prompt and asks the model to answer using them, with citations.
This lets a general model answer questions about your company's documents without retraining it.
An open-book exam: the student (the LLM) is smart, but you hand them the right pages before they answer.
Treating RAG as 'embed and vector search', then tuning prompts when answers are wrong without ever checking whether the right chunk was retrieved.
Two pipelines. Ingest (offline): parse (Layout-aware parsing), split (Chunking), attach Chunk metadata and ACLs, embed (Embedding), and write to a Vector database plus a BM25 index. Query (online): optionally rewrite the query, run Hybrid search with Metadata filtering (pre, post, filtered ANN) and ACL filters, take Top-k (retrieval k vs context k) ≈ 50 candidates, rerank to 5–8, assemble context with chunk ids, and generate with citation instructions.
Measure the two halves separately: retrieval with Recall@k and nDCG, generation with Faithfulness and answer correctness; otherwise you cannot tell which stage failed. Anthropic's contextual-retrieval write-up (2024) reported a 49% drop in failed retrievals from contextual embeddings plus BM25, and 67% with reranking added: retrieval engineering is where most of the quality lives. At scale, operations dominate: freshness via Change data capture (CDC), deletes via tombstones, Reconciliation job, tenant isolation and a latency budget per hop.
Token-based chunks of 400–800 tokens with 10–15% overlap; bge-m3 self-hosted or a hosted embedding model; BM25 plus dense fused with Reciprocal Rank Fusion (RRF) (k = 60); top 50 to a cross-encoder (bge-reranker-v2-m3 or a hosted reranker) keeping 5–8; context with chunk ids and a citation-required prompt; pgvector up to ~10M chunks, Qdrant/OpenSearch beyond; a 200-question golden set run nightly.
Developer-docs assistant: 'How do I paginate the /v2/invoices endpoint?' Hybrid search over 1.8M chunks returns 50 candidates in 35 ms; bge-reranker-v2-m3 reranks them to 6 in 140 ms; the LLM gets 3,800 tokens of context and answers with a code sample citing api-ref/invoices#list and guides/pagination. p95 end-to-end is 2.1 s, of which 1.7 s is generation.
Draw both pipelines, put a latency number on each hop, and say how each stage is measured. Commit to a default (hybrid + rerank + citations) and say what would change it. Follow-ups to expect: 'answers are wrong, where do you look?' (recall@50 on the golden set first), and 'a document was deleted, how long until the bot stops quoting it?' (deletion SLA via tombstones, proven by reconciliation).
Numbers worth knowing
- Recall@50 target
- ≥ 0.90on a golden set of real user questions; below this, the reranker and LLM cannot recover
- Retrieval latency
- 20–60 mshybrid search over 1–10M chunks with HNSW, one region
- Rerank latency
- 100–300 ms for 50 candidates568M cross-encoder on one A10G/L4, 300-token chunks
- Failed-retrieval reduction
- 49% → 67% with rerankingAnthropic contextual retrieval results, 2024
How to decide
| When | Do this |
|---|---|
| Corpus under ~100k tokens and changes rarely | Skip RAG; put it in the prompt behind a cache breakpoint. |
| Queries contain identifiers, SKUs, error codes or legal citations | Hybrid search is mandatory; dense alone misses exact matches. |
| Questions need several lookups or structured data | Escalate those to Agentic RAG or Text-to-SQL; keep single-shot as the default path. |
| Answers are wrong | Check recall@50 before touching the prompt. |
Cost maths
Cost per query ≈ LLM input + output + rerank + search. Assume $3 / $15 per M tokens (illustrative). 4k input + 300 output = $0.012 + $0.0045 = $0.0165; reranking 50 × 400 tokens on a self-hosted GPU ≈ $0.0003; search ≈ $0.0001. At 1M queries/month ≈ $17k, 97% of it the LLM. Storage: 10M chunks × 1,024 dims × 4 bytes = 41 GB of float32 vectors before index overhead.
Operating it
- Recall@50 below 0.90 on the golden set
- Vector count passes ~10M on pgvector or p95 search latency above 100 ms
- More than 10% of questions need two or more lookups
- Nightly recall@50 and nDCG@10 on the golden set: alert at −3 points
- Low-confidence rate (top rerank score under threshold): alert on +50% day over day
- Faithfulness on 500 sampled answers: alert below 0.90
- Source-vs-index document count drift: alert above 0.5%
When it goes wrong
An internal-wiki assistant kept citing an on-call escalation page that had been deleted a month earlier.
An engineer reported it; a reconciliation job then found 3% of indexed documents no longer existed at the source.
The ingest pipeline handled create and update events but ignored deletes.
Delete events written as tombstones via Change data capture (CDC), a nightly Reconciliation job diff of source ids vs index ids, and a 1-hour deletion SLA on the dashboard.
Where it shows up
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- Introducing Contextual Retrieval — Anthropic, 2024blog
Check yourself
Three questions. Every option is explained once you answer.