Context engineering
Deciding exactly what enters the model's context at each step — instructions, tools, retrieved data, history — to maximise signal per token.
Every time an AI model runs, it can only see what's in its Context window: instructions, the conversation, documents, tool results. Context engineering is choosing what goes in, and what stays out, at each step.
It goes beyond writing a good Prompt & system prompt. An AI agent that runs for dozens of steps keeps piling up text, so you decide what to summarise, what to drop, and what to fetch fresh.
Preparing a briefing pack for a busy executive: they'll read whatever you put in front of them, so the job is picking the five pages that matter, not handing over the filing cabinet.
Assuming a bigger context window removes the need to curate. Long, noisy contexts cost more, run slower and measurably lower accuracy.
Components: system instructions, tool definitions, retrieved knowledge (Context assembly), Agent memory, history and tool outputs. Models degrade with long, noisy inputs even inside the window: Liu et al. (2023) showed accuracy is lowest when the answer sits mid-context (Lost in the middle), and Chroma's "Context Rot" (2025) found performance varies with input length across all 18 models tested, even on simple tasks. Anthropic's Sept 2025 guidance frames context as a finite attention budget: find the smallest set of high-signal tokens.
Techniques: just-in-time retrieval (give the agent search tools and identifiers instead of preloading documents); compaction — summarise old turns near a threshold, keeping decisions, open issues and identifiers; tool-output trimming (return the 20 relevant lines, not the 5,000-line log); structured notes outside the context (todo files, memory); sub-agents that explore in a clean context and return summaries (Multi-agent systems); and stable-prefix ordering so Prompt caching hits. Measure it: tokens per step and task success per change.
Stable prefix (instructions → tools) first and cached; tool outputs capped at ~2–5k tokens with a truncation hint; retrieval via tools rather than preloading; compaction at ~70% of the window preserving decisions, open questions and IDs; per-step token logging with task success.
Code-search agent debugging a flaky test reads a 12,000-line CI log every turn and fails after 20 turns as context fills. Changing get_log to grep_log(pattern="ERROR|FAIL", context=5, limit=40) cuts per-turn input from ≈ 60k to ≈ 8k tokens; adding compaction at 70% of the window keeps the run under 90k; the agent finishes in 9 turns and task success on 50 replayed failures rises from 31/50 to 44/50.
Talk in context budgets: what each layer costs in tokens and what it buys in accuracy. Recommend just-in-time retrieval and trimmed tool outputs before reaching for a bigger window.
Expect: "we have a 1M window — why bother?" — cost and latency scale with input, accuracy degrades with distractors, and cache hit rates collapse if the prefix keeps changing.
Numbers worth knowing
- Models showing length-dependent degradation
- 18 of 18 testedChroma "Context Rot", 2025
- Tool response cap in Claude Code
- 25,000 tokens by defaultAnthropic, Sept 2025
- Compaction trigger
- ~70% of the windowleaves room for the next tool result and output
How to decide
| When | Do this |
|---|---|
| Tool outputs exceed ~5k tokens | Add filters, pagination and a truncation hint to the tool. |
| Runs approach the window or slow down late | Compact history; move durable facts to notes or memory. |
| Exploration would flood the main context | Delegate to a sub-agent that returns a ≤500-token summary. |
Cost maths
Per-turn input × turns × price. Code-search example at $3 / M input (illustrative, Oct 2026): before, 60k × 20 turns = 1.2M tokens = $3.60 per run (and it failed); after, 8k × 9 turns = 72k = $0.22. At 3k runs/month: $10.8k → $650, with higher success.
Operating it
- Median input per turn > 30% of the window
- Accuracy drops on runs longer than ~15 turns vs short ones
- Cache-read share of input falls below 60%
- Input tokens per step p95 (alert at 50% of the window)
- Task success vs run length bucket (alert if long runs fall >10 points below short ones)
- Cache-read share of input tokens (alert < 60%)
When it goes wrong
An internal-wiki assistant gave confidently outdated answers about the on-call policy.
Spot audit of 100 answers: 14 quoted a 2024 page superseded in 2026.
Retrieval preloaded the top 20 chunks (≈ 15k tokens); old and new pages both matched, and the stale one landed earlier in context with more text.
Just-in-time search tool returning titles + updated dates, a reranker, and a 4-chunk cap; stale answers fell to 1/100.
Related terms
Where it shows up
Sources
- Effective context engineering for AI agents — Anthropic, 2025blog
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- Context Rot: How Increasing Input Tokens Impacts LLM Performance — Chroma, 2025blog
- Writing effective tools for agents — with agents — Anthropic, 2025blog
Check yourself
Three questions. Every option is explained once you answer.