What are the trade-offs of a very large context window?
- 1 min — what a big window enables
- 3 min — the three costs with numbers
- 2 min — long context vs RAG decision
- 1 min — caps, caching, agent loop growth
A 200k–1M token window lets you put a whole contract, codebase slice or long history in the prompt. Three costs: money — every input token is billed every call unless cached; latency — TTFT grows with input (prefill is ~2 × params FLOPs per token plus quadratic attention); quality — models use long inputs unevenly (lost in the middle: accuracy drops when the relevant passage sits mid-context), and distractor passages pull answers off. Advertised window is capacity, not effective use; single-needle tests overstate multi-fact performance.
A contract-review tool sends a 150k-token master agreement plus three precedents on every question. At $3 / M input that is $0.45 per question before output; at 40k questions/month, $18k/month, and answers cite the wrong precedent on 7% of a 200-question golden set. Change: cache the master agreement as a stable prefix (reads at 0.1× = $0.045) and retrieve the top-8 relevant precedent clauses (~6k tokens) instead of whole precedents. Input cost drops to ~$0.063/question ($2.5k/month), TTFT p95 from 11 s to 3 s, wrong-precedent rate to 2%.
- Long context and RAG are complements: long context for one coherent document you must reason over whole; retrieval when the corpus is large, changing or permissioned.
- Evaluate at your real input sizes with multi-fact questions, not single needles; plot accuracy vs position and vs context length.
- Context engineering — what goes in, order, compression — usually beats buying a bigger window. Put instructions and the question at the end, stable content first so it caches.
- Org scope: the platform team sets context caps per product tier and exposes cache-hit rate per route; product teams justify any route above 32k with an eval. Rollout: raise caps behind a flag per tenant.
- Where it breaks: agent loops that append every tool result forever — input grows per step, so cost per task grows quadratically with step count. Summarise or prune at a token budget.
assume $3 / M input and $15 / M output tokens (illustrative frontier list prices, Oct 2026); cache write 1.25× input for a 5-minute TTL, cache read 0.1×. Uncached: 150k × $3 / M = $0.45/question. Cached prefix: 150k × $0.30 / M = $0.045, plus a cache write of 150k × $3.75 / M = $0.56 once per 5-minute TTL window per active document. Retrieval adds 6k × $3 / M = $0.018. At 40k questions/month on ~2k distinct documents, writes ≈ $1.1k and reads+retrieval ≈ $2.5k.
What the answer sounds like at each level
Knows bigger context costs more and is slower, and that models miss things in the middle.
Quantifies cost and TTFT at real sizes, uses caching for stable prefixes, and evaluates position sensitivity.
Sets per-tier context caps, makes long-context routes earn their place with evals, and designs prompts for cache hits.
Follow-ups the interviewer will push
1Now traffic is 10×. What changes?
Cache-hit rate becomes the cost lever: 400k questions × $0.45 uncached is $180k/month vs ~$25k with a stable cached prefix. Also check provider input-token-per-minute limits, which bind before request limits on long prompts.
2Self-hosting a 70B model with 128k context — what's the constraint?
KV cache: 40 GB per full-length session in BF16. Two H100s with FP8 weights fit only a handful of 128k sessions; route long-context traffic to a separate pool.
Red flags interviewers listen for
- Says 'just put everything in context' without cost or quality numbers.
- Trusts needle-in-a-haystack scores as proof of long-context reasoning.
- Lets agent context grow without a budget.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Cost/latency | Works the per-call input cost and the effect of caching. |
| Quality | Cites position effects and distractors, and evaluates at real sizes. |
| Design | Chooses long context vs retrieval with a stated rule and prompt ordering for caching. |
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- Prompt caching — Anthropic, 2025docs