Precision
Of the results you returned, the fraction that are relevant. Precision@k measures how much of the top k is useful versus noise.
Precision asks: of what you returned, how much was actually relevant? If 2 of your top 5 results are useful, precision@5 is 0.4. In Retrieval-Augmented Generation (RAG), low precision means the prompt is full of irrelevant text that costs money and can distract the Large Language Model (LLM).
Fishing again: precision is how much of what is in the net is fish rather than seaweed.
Reporting precision without recall (or the reverse). Either is trivially maximised alone: return one sure thing for precision, return everything for recall.
precision@k = |relevant ∩ top-k| / k. It trades against Recall@k: larger k usually raises recall and lowers precision. For classifiers (guardrails, intent routers, PII detectors), precision = TP / (TP + FP) and pairs with recall = TP / (TP + FN); F1 is their harmonic mean, and Fβ weights recall β times as much. For ranked retrieval, precision@k ignores order inside the top k; average precision (AP) and Context precision reward relevant items appearing earlier. For RAG specifically, precision of the assembled context matters more than precision@100 of the first stage, because the reranker exists to clean that up.
For retrieval: report precision@k at the final k alongside recall@k and nDCG@10. For classifiers (guardrails, routers): report precision and recall at the chosen threshold plus the PR curve, never accuracy alone on imbalanced data.
E-commerce catalog search for 'waterproof trail running shoes women size 8'. Top 5: (1) women's waterproof trail shoe ✓, (2) men's waterproof trail shoe ✗, (3) women's trail shoe, not waterproof ✗, (4) women's waterproof trail shoe ✓, (5) waterproof hiking sock ✗. precision@5 = 2/5 = 0.4. Adding a gender facet filter from query parsing removes (2) and (5); the next two results are relevant, so precision@5 rises to 0.8 with no ranking-model change.
Say: in RAG I watch precision of what reaches the prompt (context precision) and recall at the first stage, because the pipeline is designed to trade them across stages. For classifiers I pick the operating point from the cost of a false positive vs a false negative, not from F1. Expect: 'your PII detector has 0.99 precision, is that good?' Not without recall: it may be missing half the PII.
| Rank | Result | Relevant | Precision@rank |
|---|---|---|---|
| 1 | W waterproof trail | ✓ | 1/1 = 1.00 |
| 2 | M waterproof trail | ✗ | 1/2 = 0.50 |
| 3 | W trail, not waterproof | ✗ | 1/3 = 0.33 |
| 4 | W waterproof trail | ✓ | 2/4 = 0.50 |
| 5 | Waterproof sock | ✗ | 2/5 = 0.40 |
Numbers worth knowing
- Final-context precision target
- ≥ 0.65–8 chunks, single-hop Q&A
- Guardrail precision floor
- ≥ 0.95blocking classifier on user traffic; false blocks anger users
- Token waste
- (1 − precision) × context tokense.g. 0.4 precision on 4k tokens = 2.4k wasted tokens/query
How to decide
| When | Do this |
|---|---|
| False positives are costly (blocking legit users) | Optimise precision; accept lower recall, add human review |
| Misses are costly (PII leak, safety) | Optimise recall; route uncertain cases to review |
| Retrieval precision low but recall high | Add a reranker or metadata filters, not a bigger model |
Cost maths
Wasted spend = (1 − precision@k) × k × tokens per chunk × price × queries. 8 chunks × 450 tokens, precision 0.4 → 2,160 wasted tokens × $3/M = $0.0065/query; at 2M queries/month ≈ $13,000. Raising precision to 0.75 with a reranker saves ~$7,500/month.
Operating it
- Final-context precision < 0.5 on the golden set
- Guardrail false-block complaints > 0.5% of sessions
- Prompt token spend growing faster than traffic
- Final-context precision on nightly golden set; alert < 0.5
- Classifier precision and recall on a labelled weekly sample; alert on either dropping > 5 points
- Retrieved-but-uncited chunk rate (proxy for low precision); trend
When it goes wrong
A prompt-injection classifier launched with 99% precision; three weeks later an injected document exfiltrated a summary
Red-team replay of the incident: the classifier flagged 52% of known injection samples
The launch review looked only at precision on benign traffic; recall on attacks was never measured
Eval set with attack and benign samples, both precision and recall reported, threshold chosen for ≥ 0.9 recall with human review on flags
Related terms
Sources
- Introduction to Information Retrieval (ch. 8, Evaluation) — Manning, Raghavan & Schütze, 2008book
Check yourself
Three questions. Every option is explained once you answer.