Lost in the middle
2023 models used mid-context facts worse than edge ones. In 2026, single facts are found; multi-hop reasoning and aggregation over long context still degrade.
In 2023 researchers found that if you put the key fact in the middle of a long Prompt & system prompt, models used it less than if it was at the start or end. Newer models mostly fixed that for finding one fact. What still breaks is harder work over long inputs: connecting several facts spread across the text, or counting and summarising across all of it. So the advice holds: give the model fewer, better passages and put the best ones where it reads most carefully.
Finding one name in a phone book is easy at any page. Working out how many people on three different pages share a street, and who moved in first, is where you start making mistakes.
Testing long-context readiness with single-needle retrieval. A model that finds one planted sentence at 200k can still fail a question that needs three facts from different sections.
Liu et al. (2023) moved a gold passage through 10–30 retrieved documents and measured QA accuracy: a U-shape, best at position 1, next at the end, worst in the middle; with 20 documents GPT-3.5-Turbo in the middle scored below its own closed-book baseline. Those were 2023 models with 4k–16k effective windows. By 2025–26, frontier models score near 100% on vanilla needle-in-a-haystack at 128k+, so 'the model cannot see the middle' is no longer the right framing.
The real degradation, shown by RULER (Hsieh et al. 2024), is task-dependent: models near-perfect on single-needle retrieval fall off on multi-key and multi-value retrieval, multi-hop tracing (follow a variable through a chain of assignments) and aggregation (most frequent words across the whole context). Of the models RULER evaluated, all claiming 32k+ windows, only about half held satisfactory performance at 32k. The effective context for reasoning is shorter than the advertised window, and position effects reappear as soon as the task needs several facts at once. Practical response unchanged: rerank and send fewer chunks, put the strongest first and second-strongest last, restate the question after long context, and split multi-hop questions with Query decomposition.
Rerank to 5–10 chunks; order best-first, second-best-last; restate the question after the context block; decompose multi-hop questions into single-hop sub-queries. Before raising context limits, run a 50-question sweep on your own model varying gold position (1, 25%, 50%, 75%, end) and facts-needed (1, 2, 4).
Financial-filings assistant, Oct 2026 frontier model, 120k tokens of three 10-Ks. 'What was FY2025 revenue for segment B?' is answered correctly 49/50 times wherever the table sits. 'Which segment grew fastest across 2023–2025, and did its operating margin rise?' needs six numbers from three filings: 31/50 correct with full filings, 44/50 when the same model gets 9k tokens of the reranked segment tables only, best-first.
Say: the 2023 U-curve was a retrieval failure that frontier models have mostly closed; the 2026 failure is reasoning over many scattered facts, which RULER-style multi-hop and aggregation tests expose. So I still retrieve and rerank instead of stuffing 500k tokens, and I measure on my own task with position and fact-count sweeps. Expect: 'Why not just use the 1M window?' Answer: cost and latency scale with tokens, and accuracy on multi-fact questions falls with length even when single-fact lookup does not.
| RULER task family | What it tests | Long-context behaviour |
|---|---|---|
| Single needle (NIAH) | Find one planted fact | Near-ceiling for frontier models |
| Multi-key / multi-value | Find several facts | Drops with length |
| Multi-hop tracing | Follow a chain of refs | Drops sharply |
| Aggregation | Count / summarise all | Drops sharply |
| QA with distractors | Answer amid noise | Drops with length |
Numbers worth knowing
- Middle-position penalty (2023)
- > 20 accuracy pointsGPT-3.5-Turbo, 20 documents, middle vs first position (Liu et al. 2023)
- Models holding up at 32k
- about halfRULER, models all claiming ≥ 32k windows (Hsieh et al. 2024)
- Single-needle retrieval
- ≈ 95–100%frontier models at 128k+, Oct 2026; does not predict multi-hop accuracy
How to decide
| When | Do this |
|---|---|
| Question needs one fact | Long context is fine; still rerank to control cost and latency |
| Question needs 3+ facts from different places | Decompose, retrieve per sub-question, assemble a short context |
| Task aggregates over the whole corpus (count, trend) | Map-reduce over chunks or structured extraction to a table, not one long prompt |
Cost maths
Stuffing vs retrieving at $3/M input (illustrative, Oct 2026): 120k-token prompt = $0.36/query; reranked 9k = $0.027/query, 13× cheaper. At 50k queries/month: $18,000 vs $1,350, and the shorter prompt scored higher on multi-fact questions in the example above. Time-to-first-token also scales roughly linearly with prefill length.
Operating it
- Product wants to raise context from 32k to 200k: run the position × fact-count sweep first
- Multi-hop slice of the golden set falls > 10 points below single-hop
- Prompt p95 tokens grows quarter over quarter without a matching accuracy gain
- Accuracy by prompt-length bucket (< 8k, 8–32k, 32–128k); alert if the top bucket trails the bottom by > 10 points
- Accuracy by facts-needed tag in the golden set; alert on a > 5-point drop after a model change
- Prompt tokens p95; alert on a > 50% week-over-week rise
When it goes wrong
A contract-review tool missed indemnity carve-outs in long master agreements but never in short ones
Lawyer feedback tags clustered on contracts over 80 pages; a length-bucketed golden-set slice confirmed accuracy 91% under 30 pages, 64% over 80
The whole agreement was sent in one prompt; carve-outs required linking a definition in section 1 to exceptions in an exhibit, a multi-hop task that degraded with length
Clause-level retrieval with definition expansion, 12k-token assembled context, and a second pass that checks every defined term used in the cited clauses
Related terms
Where it shows up
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
- RULER: What's the Real Context Size of Your Long-Context Language Models? — Hsieh et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.