What causes an LLM to ignore retrieved context?
- 1 min — diagnose: was the right chunk in context?
- 3 min — five causes and their fixes
- 1 min — measure context precision and faithfulness separately
Several distinct causes, and they need different fixes:
- Lost in the middle. Attention favours the start and end of long contexts. Fix by ordering — best evidence first and last.
- Parametric conflict. The model's training says one thing, your document says another; it trusts itself. Fix with explicit instruction precedence in the prompt.
- Noise dilution. Too many weak chunks drown the strong one. Fewer, better chunks beat more chunks.
- Weak prompting. No instruction to ground, no instruction to refuse when unsupported.
- Formatting. Mangled tables and broken markdown in parsed PDFs are hard to read.
A classic: a model answering with the old pricing it learned in pretraining while the current price sheet sits in context. The fix was a prompt line stating that retrieved documents override prior knowledge, plus moving the price table to the end of the context.
Separate retrieval failure from generation failure before debugging. Measure context precision (did we retrieve the right thing) and faithfulness (did the answer follow it) independently. Teams waste months tuning retrieval for a prompting bug. Org scope: that split should be institutional — the retrieval team owns context precision, the application team owns faithfulness, and both metrics sit on one dashboard so a regression lands with the right owner on day one.
What the answer sounds like at each level
The model may ignore context if the prompt doesn't tell it to use the documents.
Lists the distinct causes (position, parametric conflict, noise, prompting, formatting) with a fix for each.
Separates retrieval from generation failure with an oracle test and two metrics before debugging, and turns the fix into a standing groundedness metric.
Follow-ups the interviewer will push
1Now you switch to a newer model with a 1M-token window. Does lost-in-the-middle go away?
It shrinks on newer models but does not vanish, and noise dilution gets worse as you add more. Keep top-k small after reranking and re-run a position-sensitivity test (move the gold chunk through positions 1..N) on the new model.
2Now it's a regulated domain where a stale answer is a compliance incident.
Add a post-generation groundedness check that blocks answers containing claims or entities absent from the context, and fall back to abstention plus a citation to the source document.
Red flags interviewers listen for
- Jumps straight to retrieval tuning without checking whether the right chunk was in context.
- Treats all 'ignored context' cases as one bug.
- Assumes bigger context windows fix it.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Diagnosis | Uses traces or an oracle test before changing anything. |
| Breadth | Distinguishes position, parametric conflict, noise and formatting. |
| Measurement | Leaves behind a standing metric, not a one-off fix. |
The answers were right, then they weren't
Illustrative exampleOur customer-facing docs assistant started quoting a deprecated API parameter that we had removed from the documentation months earlier. Support escalated it as a retrieval bug.
I owned the retrieval layer, so it landed on me, but I didn't think it was retrieval.
Before touching the index I pulled 40 failing traces and checked whether the correct chunk was in context. It was, in 37 of 40 — the model was overriding it with pretrained knowledge of our older public API. I ran a small A/B: one arm added an explicit precedence instruction and a refusal clause, the other reordered context so the authoritative chunk sat last. Both helped; together they took the contradiction rate from 9% to under 1% on our eval set. I also added a groundedness check that flags answers containing entities absent from the context, which gave us a standing metric instead of waiting for support tickets.
Fixed in under a week with no index changes. The broader win was process: we split our eval dashboard into retrieval metrics and generation metrics, and three later "retrieval bugs" turned out to be prompt issues too.
Sources
- Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper