How do you decide whether a problem actually needs RAG?
- 1 min — the three questions
- 2 min — worked cost comparison: long context vs RAG
- 1 min — structured-data questions → text-to-SQL
- 1 min — revisit threshold
Three questions. (1) Is the knowledge outside the model and does it change? (2) Is the corpus bigger than a reasonable context window — and is cost at QPS prohibitive? (3) Do answers need citations? If no to all three, you don't need RAG.
A 60-page product manual answered 500 times a day: just put it in the prompt with caching. A 4-million-document legal archive: RAG. A SQL-answerable question ("how many orders shipped late last month"): neither — that's text-to-SQL, and retrieving prose about orders will give confidently wrong numbers.
The strongest answer here is a refusal. "The highest-leverage thing I've done on RAG projects is talk two of them out of existence." Long context plus prompt caching has moved the threshold considerably; re-check the arithmetic rather than assuming the 2023 answer still holds. Org scope: make this a design-review gate owned by the platform team, with a one-page threshold memo teams reuse, so the default answer to 'we need a vector DB' is a measured spike rather than a procurement.
Assume $3 / M input tokens, Anthropic-style cache pricing (write 1.25×, read 0.1×, 5-minute TTL) — illustrative list prices, Oct 2026.
- 60-page manual ≈ 30k tokens.
- Cached read: 30k × $0.30/M = $0.009 per query. Cache write: 30k × $3.75/M = $0.11, paid when the 5-minute TTL lapses.
- 500 queries/day over 10 business hours ≈ 0.8/min → cache stays warm: ≈ $4.50/day + a few writes.
- RAG at 4k tokens/query, uncached: $0.012 per query ≈ $6/day, plus a vector store and ingestion to operate.
Verdict: long context wins at this size; revisit at roughly 150k tokens of corpus or when ACLs differ per user.
What the answer sounds like at each level
Uses RAG when the documents don't fit in the prompt.
Applies the three questions (external and changing, size vs context and cost, citations) and names text-to-SQL as a different problem.
Does the arithmetic live with cached long context vs RAG, sets a revisit threshold, and is comfortable recommending no new infrastructure.
Follow-ups the interviewer will push
1Now usage grows from 500 to 50,000 queries a day.
Re-run the maths: at 30k cached tokens per query, cache reads are ~$0.009 each, so $450/day. A RAG pipeline at ~4k tokens per query is ~$0.012 at full input price — similar per query but now with infra to run. Long context still wins until the corpus outgrows the window or latency on 100k+ tokens hurts.
2Now different users may only see different documents.
Per-document ACLs break whole-corpus stuffing, because every user would need a different cached prefix. That is the strongest argument for retrieval with ACL filters, independent of size.
Red flags interviewers listen for
- Assumes every LLM product needs a vector database.
- Routes aggregate/numeric questions to prose retrieval.
- Quotes 2023 context limits as current.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Judgement | Willing to say 'no RAG' with evidence. |
| Quantitative | Computes per-query cost including cache read/write multipliers. |
| Scope | Recognises ACLs and corpus size as the real triggers. |
Killing a RAG project
Illustrative exampleA product team asked my platform group to stand up a vector pipeline for their onboarding assistant. They had budget approved and a vendor shortlist.
I was the reviewing engineer on the design. My job was to sign off on infra, but the design didn't justify itself.
I counted the corpus: 340 pages, changing a few times a quarter. I built a two-day spike that stuffed the whole thing into context with prompt caching and ran both approaches against 120 questions the support team had collected. Flat context scored higher on faithfulness and cost less per query at their projected volume. I wrote a one-page memo with both numbers and a threshold — "revisit this when the corpus passes roughly 1,500 pages or we need per-document ACLs."
They shipped six weeks earlier with no vector infrastructure to operate. The threshold memo got reused by two other teams. About a year later one of them did cross the line and came back with the memo in hand, which made that review trivial.
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper