Context window
The maximum number of tokens, prompt plus output, a model can handle in one call. Anything outside it does not exist for the model.
The context window is the model's working memory for one request. The instructions, the conversation so far, any retrieved documents and the answer it writes all have to fit inside it, measured in tokens.
If something is not in the window, the model cannot see it. That is why Retrieval-Augmented Generation (RAG) exists: to pick the few most useful passages and put them into this limited space.
A desk of fixed size: you can only work with the papers that fit on it, and a cluttered desk makes the important page harder to find.
Believing a bigger window removes the need for retrieval. Long prompts are slower, pricier, and on most models less accurate than a small, well-chosen context.
As of Oct 2026, frontier APIs offer 200k–1M+ token windows and most open-weights models 128k (Llama 3.1: 128k). Output limits are separate and smaller: 64k–128k+ tokens on frontier models, less on many open models; reasoning tokens usually count against the output limit. These numbers move every quarter.
The limit is economic as much as architectural. Prefill compute grows with prompt length (quadratically in Attention), and the KV cache grows linearly: Llama-3.1-70B holds 320 KB per token in BF16, so one 128k request needs 40 GB of KV. Long prompts therefore cost more money, more TTFT and fewer concurrent users.
Advertised length is not usable length. RULER (2024) showed most models fall well below their short-context accuracy before reaching their advertised window, and Lost in the middle shows recall depends on position. Treat the window as a budget: system prompt, tool schemas, history, retrieved context and reserved output each get an allocation, and Context assembly plus reranking decide what earns a slot.
Explicit per-route budget: system + tools ≤ 5k, history capped at 8k with older turns summarised, retrieved context 3–8k, reserved output from measured p99 + 30%. Set server-side max_model_len (vLLM) to the p99 real context, not the model maximum. Put stable content first for Prompt caching.
A clinical-intake assistant on a 200k-window model budgets: system prompt and tool schemas 4k, patient history summary 3k, 6 retrieved guideline chunks × 700 = 4.2k, conversation 5k, reserved output 2k: ~18k total. The alternative, pasting the whole 180k-token formulary into every call, costs $0.54 instead of $0.054 per call at an assumed $3 / M input, pushes TTFT from ~1 s to 10–20 s, and makes dosage recall depend on where in the formulary the answer sits.
Asked 'we have a 1M-token window, why retrieve?', answer with five things: per-call cost scales linearly with prompt tokens, TTFT scales with prefill, recall degrades with depth, retrieval gives freshness without re-sending everything, and retrieval enforces per-user access control. Then commit: stuff the context only when the corpus is under ~100k tokens, rarely changes and is reused enough to hit Prompt caching. Expect the follow-up 'how would you prove long context is worse here?': run needle tests at 25/50/75/100% depth on your own documents.
Numbers worth knowing
- Frontier input window
- 200k–1M+ tokenshosted APIs, Oct 2026; open-weights models commonly 128k
- Frontier output limit
- 64k–128k+ tokensOct 2026, quarterly volatility; reasoning tokens usually count against it
- KV cache per token
- 320 KBLlama-3.1-70B, BF16 (80 layers × 8 KV heads × 128 dims × 2 × 2 bytes); 128k context ≈ 40 GB
- Prompt cost scaling
- linear in input tokensa 180k-token call at $3 / M = $0.54; some providers charge a higher rate above 200k input tokens
How to decide
| When | Do this |
|---|---|
| Reference material under ~100k tokens, rarely changes, reused across many calls | Put it in the prompt once, behind a cache breakpoint; skip retrieval. |
| Corpus is large, changes daily, or has per-user permissions | Use Retrieval-Augmented Generation (RAG) with a 3–8k token retrieved-context budget. |
| Conversations run past ~20 turns | Summarise turns older than the last 6 into a rolling 1–2k token summary. |
| Task genuinely needs whole-document reasoning (summarise a 300-page contract) | Use the long window, but test recall at multiple depths and cache the document. |
Cost maths
Input cost per call = prompt_tokens × p_in. Assume $3 / M input (illustrative, Oct 2026). 18k-token budgeted call = $0.054; 194k-token stuffed call = $0.58. At 50k calls/day: $2.7k/day vs $29k/day. With Prompt caching the stuffed version drops toward 0.1× on the cached document but still pays the 1.25× write each time the 5-minute cache expires, and still pays full TTFT on misses.
Operating it
- Input tokens p95 above 70% of the window on any route
- Needle-recall at 50% depth below 90% on your own documents
- TTFT p95 above SLO and growing with conversation length
- Prompt tokens p95/p99 per route as % of window: alert above 70%
- Context-length-exceeded errors: alert above 0.1%
- TTFT p95 bucketed by prompt length: alert when the >50k bucket breaches SLO
When it goes wrong
A contract-review agent told a lawyer an MSA had 'no limitation of liability cap'; the cap existed.
The lawyer escalated; replaying the trace showed the whole 190k-token agreement in context and the cap clause at token ~140k.
The team relied on the long window instead of retrieval; on internal needle tests the model's recall at that depth was 78%, and nobody had run those tests before launch.
Section-aware Chunking and clause retrieval for question answering, the full document only for summarisation, and needle tests at 50k/100k/150k depth added to the Eval gate (CI).
Related terms
Where it shows up
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- RULER: What's the Real Context Size of Your Long-Context Language Models? — Hsieh et al., 2024paper
- Context windows — Anthropic, 2026docs
Check yourself
Three questions. Every option is explained once you answer.