Abstention & relevance thresholds
The system declining to answer when evidence is missing or confidence is low, trading coverage for accuracy on what it does answer.
Sometimes the right answer is 'I don't know' or 'I couldn't find that in the documents'. Abstention means building the system so it says that on purpose, instead of guessing. You decide a rule, such as 'if no retrieved passage scores above a threshold, don't answer', and you measure how often it abstains and how often its answers are right.
A pharmacist who says 'let me check with your doctor' instead of guessing a dose.
A golden set with no unanswerable questions. You cannot tune abstention, or even notice its absence, if every test question has an answer in the corpus.
Abstention can fire at three points. Pre-generation: no chunk above a reranker-score floor, or intent classified as out-of-scope; cheapest and most reliable. In-generation: the prompt allows 'not found in the sources' and few-shot examples show it; models comply far better when the refusal path is explicit. Post-generation: a verifier finds unsupported claims, or self-reported or sampled-consistency confidence is low. Kadavath et al. (2022) showed larger models are reasonably calibrated on whether they know an answer, but calibration degrades out of distribution, so retrieval-score floors are the more robust signal. The trade-off is a selective-prediction curve: accuracy-on-answered rises as coverage falls. Pick the operating point from the cost of a wrong answer vs an unhelpful one, and always give the abstention a next step (link, human handoff, rephrase hint).
Golden set with 10–15% unanswerable items. Pre-generation gate: top reranker score below a tuned floor → decline without calling the LLM. Prompt explicitly allows 'I couldn't find this in the sources' with one example. Every decline includes a next action (search link, human handoff, suggested rephrase). Log declines with the top score for tuning.
Developer-docs assistant, 600 golden questions including 80 deliberately unanswerable. With no abstention: 100% coverage, 82% accuracy, and 71 of the 80 unanswerable got confident fabrications. With a reranker floor of 0.35 plus 'say not found' instructions: 81% coverage, 93% accuracy on answered, 68 of 80 unanswerable correctly declined, and 34 answerable questions wrongly declined. The team tunes the floor to 0.30, accepting 86% coverage at 91% accuracy, and adds 'search the docs' and 'ask in #dev-help' links to every decline.
Say: I put the cheap gate first (retrieval-score floor plus out-of-scope intent), make refusal an explicit allowed output, and choose the threshold from a coverage-accuracy curve on a golden set that contains unanswerable questions. Expect: 'how do you know you aren't over-refusing?' Track false-abstention rate on answerable items and the share of declines users rephrase and get answered.
Numbers worth knowing
- Unanswerable share in golden set
- 10–15%enough to measure false-answer rate with ±5-point CI at n ≈ 500
- Typical operating point
- 85–92% coverageinternal Q&A where wrong answers are costly but not regulated
- Cost saved per pre-gen decline
- 100% of LLM cost for that querygate fires before generation
How to decide
| When | Do this |
|---|---|
| Wrong answer has legal, medical or financial cost | Lower coverage (70–85%) and route declines to a human |
| Casual internal tool, wrong answers are cheap | Higher coverage (90–95%) with visible sources |
| Many declines on answerable questions | Fix retrieval recall before lowering the floor |
Cost maths
Expected cost per query = P(wrong) × cost_wrong + P(decline) × cost_decline. Assume a wrong benefits answer costs $40 in rework and a decline costs $3 of an HR agent's time. No abstention: 0.18 × 40 = $7.20. Floor 0.30: 0.86 × 0.09 × 40 + 0.14 × 3 = $3.10 + $0.42 = $3.52. Abstention halves the expected cost.
Operating it
- Decline rate > 25% of traffic: retrieval or corpus coverage problem, not a threshold problem
- Corpus or reranker model changes: re-sweep the floor
- Human handoff queue > staffing capacity: tighten scope or improve recall
- Decline rate; alert if it moves > 5 points from the 7-day baseline
- False-abstention rate on answerable golden items; alert > 10%
- Fabrication rate on unanswerable golden items; alert > 10%
When it goes wrong
After a reranker upgrade, an internal IT assistant suddenly refused 40% of questions
Decline-rate dashboard jumped from 9% to 41% within an hour of deploy
The new reranker's scores were on a different scale; the fixed 0.35 floor now cut most valid answers
Rolled back, re-swept the floor on the golden set for the new model (0.12), stored the floor alongside the model version, and added a canary check on decline rate
Related terms
Where it shows up
Sources
- Language Models (Mostly) Know What They Know — Kadavath et al., 2022paper
- Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.