Context precision
RAG metric: are the relevant chunks ranked at the top of the context? Rank-weighted precision, so noise above the evidence costs more.
Context precision looks at the chunks you gave the model and asks whether the useful ones come first. If the first chunk is relevant and the useful ones are near the top, the score is high. If the evidence is buried under irrelevant chunks, the score drops, even if the evidence is technically there.
A briefing pack for a meeting: it is better if the two pages that matter are on top, not pages 7 and 9 after five pages of background.
Optimising context precision by sending one chunk. The score rises while context recall and multi-hop accuracy collapse; always report it with context recall.
Ragas definition: context precision@K = Σ_{k=1..K} (precision@k × v_k) / (number of relevant items in the top K), where v_k = 1 if the chunk at rank k is relevant. It is average precision over the assembled context, so each relevant chunk is credited with the precision at its position. Relevance is judged per chunk by an LLM against the question and a reference answer (or, reference-free, against the generated answer). It diagnoses the reranker and context assembly, complementing context recall (did we include all needed evidence?) and Faithfulness (did generation stick to it?). Because modern models handle a few distractors well on single-hop questions, low context precision costs more in tokens and in multi-hop accuracy than in single-fact accuracy; prioritise it accordingly.
Ragas context_precision (reference-based when the golden set has reference answers), computed on the assembled context, not the first-stage list; reported with context recall and faithfulness as a triad, per slice.
Insurance claims assistant, question: 'Was the roof claim filed within the policy's reporting window?'. Five chunks: (1) reporting-window clause ✓, (2) roof-material exclusions ✗, (3) claim filing timestamp ✓, (4) adjuster bio ✗, (5) generic FAQ ✗. precision@1 = 1, precision@3 = 2/3. Context precision = (1 + 0.667) / 2 = 0.83. Swapping chunks 2 and 3 gives (1 + 1) / 2 = 1.0; dropping 4 and 5 does not change the score but saves ~900 tokens.
Say: context precision tells me whether the reranker and assembler put evidence first; context recall tells me whether evidence was found at all; faithfulness tells me whether generation used it. I look at all three before touching the prompt. Expect: 'why not plain precision@k?' Because it ignores order, and order is what the reranker controls.
| Rank | Chunk | Relevant | precision@k (if relevant) |
|---|---|---|---|
| 1 | Reporting-window clause | ✓ | 1/1 = 1.00 |
| 2 | Roof exclusions | ✗ | — |
| 3 | Filing timestamp | ✓ | 2/3 = 0.67 |
| 4 | Adjuster bio | ✗ | — |
| 5 | Generic FAQ | ✗ | — |
| Context precision | (1.00 + 0.67) / 2 = 0.83 |
Numbers worth knowing
- Healthy range
- ≥ 0.755–8 chunk contexts after a cross-encoder reranker
- Typical without reranker
- 0.45–0.65bi-encoder top-8 straight into the prompt
- Judge calls per item
- one per chunk8 chunks = 8 small-model calls, ~$0.002 total
How to decide
| When | Do this |
|---|---|
| Context precision low, context recall high | Improve reranker or reorder; consider fewer chunks |
| Both low | First-stage retrieval problem: fix before tuning order |
| Both high, faithfulness low | Generation problem: prompt, model or decoding |
Cost maths
Golden-set run: 500 items × 8 chunks × ~600 tokens judge input at $0.80/M (illustrative) ≈ $1.90 per run. Cheap enough for every PR tier.
Operating it
- Context precision < 0.6 on any major slice
- Multi-hop slice accuracy trails single-hop by > 10 points while context recall is fine
- Prompt tokens per answer growing without a quality gain
- Context precision, context recall and faithfulness nightly; alert on any > 0.05 drop
- Average rank of first relevant chunk in assembled context; alert if > 2
When it goes wrong
Legal research answers became vaguer after a switch to larger chunks, with no drop in retrieval recall
Eval triad: context recall flat at 0.88, context precision 0.79 → 0.52, answer correctness −7 points on multi-hop questions
1,500-token chunks each contained one relevant paragraph and several irrelevant ones; the judge marked most chunks partially irrelevant and the model hedged
Back to 400-token chunks with parent-section expansion only for the top 2; context precision returned to 0.77
Where it shows up
Sources
- Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023paper
- Introduction to Information Retrieval (ch. 8, Evaluation) — Manning, Raghavan & Schütze, 2008book
Check yourself
Three questions. Every option is explained once you answer.