Talking a team out of RAG (when it is the wrong tool)

The design doc proposes a full RAG stack for an internal engineering-standards assistant: an ingestion pipeline, a vector database, a chunking strategy, hybrid retrieval and a reranker. The corpus is 310 pages of standards and runbooks, about 150,000 tokens, edited a few times a week. Expected traffic is 2,000 questions a day from 600 engineers, all of whom may read every page. Six weeks of work and a new on-call rotation, for a corpus that fits inside a single prompt with room to spare.

A Staff-level contribution here is not “RAG is overkill”. It is a comparison the team can check, a decision that names its own expiry, and a plan that keeps the RAG path cheap to reach later.

• The corpus fits in context and is shared. Under ~30–50% of the model's context window, read by everyone, prefix reused within the cache TTL. Retrieval misses disappear as a failure class because nothing is retrieved.
• The question is about rows, not prose. “Incidents by service last quarter” is an aggregate. Chunked spreadsheets return a sample of rows and the model guesses the total. Text-to-SQL against a governed schema returns the exact number and an auditable query.
• There are 30 questions, not 30,000. A curated FAQ behind an intent classifier beats any retrieval system on accuracy, latency and cost.

Assume $3 per million input tokens and $15 per million output tokens (illustrative list prices, Oct 2026); Anthropic-style cache pricing of 1.25× base for a 5-minute cache write and 0.1× for a cache read (some current models price reads lower, e.g. 0.05× on Claude Opus 5.5); 2,000 questions spread over a 10-hour working day, so one every ~18 seconds and the 5-minute cache stays warm; ~10 cache writes a day after idle gaps and corpus edits; RAG sending 8,000 tokens of retrieved context. Output is ~400 tokens per answer in every design (~$12/day) and is left out below.

Caching turns a $27k/month design into a $2.9k/month one. RAG is still cheaper on model spend by ~$1.5k a month, which is less than one engineer-week of building and operating the pipeline. At 20× the traffic the gap is ~$25k a month and RAG wins outright. That is the shape of the argument: not “which is better” but “below what volume is the simpler design cheaper in total”.

Prefill cost grows with input length, and a 150k-token prompt is a lot of prefill. Assumptions, to be replaced by your own measurements: uncached prefill at 10–20k tokens/second on a hosted frontier model; cached prefix reads that skip almost all of that work; retrieval (hybrid plus a small reranker) at 100–200 ms.

Two consequences. Cached long context is only fast on the warm path, so your p99 is set by cache misses: pre-warm the cache after every corpus edit and on a timer before the TTL lapses, or use the 1-hour TTL (2× write cost) for bursty traffic. And never put anything per-request (a timestamp, the user's name) before the cached prefix, or every request is a miss.

The scenario above was chosen so that every engineer may read every page. Most real corpora are not like that, and per-document access control is where long-context designs most often break.

Rule one: anything in the prompt is visible to the user. You cannot put a document the user may not read into the context and instruct the model not to reveal it. Prompts leak through paraphrase, summaries and prompt injection. Filtering must happen before the model sees anything, and it must fail closed: if the permission lookup errors, the user gets fewer documents, never more.

Rule two: count the distinct permission sets, not the documents. A cached prefix is shared only by requests with byte-identical prefixes, so each distinct set of visible documents needs its own prefix and its own cache writes.

• 1 set (everyone sees everything): one prefix, the economics above hold.
• 3–10 sets (e.g. public, engineering, security, finance): one prefix per group, ordered so the shared base comes first. Cost and warm-cache logistics scale by the group count; often still cheaper than RAG at modest volume.
• Hundreds or thousands of sets (per-user sharing, per-client matters in a law firm, per-patient records): caching collapses, every request is effectively uncached, and RAG with ACL filtering at retrieval time is the right design.

Rule three: revocation has a latency, and you must state it. In a long-context design, revoking access means rebuilding that group's prefix; the window is however long until the next rebuild. In RAG, the index's copy of each document's ACL lags the source system by the sync interval. For regulated corpora, re-check permissions against the source of truth at query time for the final 5–20 chunks, and log the decision. In multi-tenant products, keep tenants in separate caches and indexes, not just separate filters.

The most useful artefact is a written condition under which the decision flips, owned by a named team and checked on a dashboard:

It turns a debate into a monitor, tells the RAG advocates exactly when they win, and forces the golden set and a small RAG baseline to stay alive, so the switch is a measured move rather than a rewrite under pressure. Tie it to the SLO review, not a doc nobody rereads.

Frame it as a trade-off with a crossover point. The engineers proposing RAG are right that it scales further; agree with that first, then show today's numbers and the volume, corpus size or ACL count at which their design wins. Offer to build the golden set and the document registry first, because both designs need them, and the RAG path stays one sprint away instead of six weeks.

Mixed answers are normal: route metric questions to text-to-SQL, keep the small shared handbook in a cached prefix, and retrieve only over the large, permissioned, fast-changing archive. Three simple paths beat one pipeline that does everything badly.