Hallucination
When an LLM states something fluent and confident that is false or not supported by its sources.
A hallucination is when the model makes something up: a fake citation, a wrong price, a policy that does not exist, delivered in the same confident tone as a correct answer.
It happens because the model is built to produce plausible text, not to check facts. Systems reduce it by giving the model the right documents (Grounding & groundedness checks), requiring it to cite them, and allowing it to say 'I don't know' (Abstention & relevance thresholds).
A confident dinner guest who would rather invent a plausible answer than admit they don't know.
Assuming RAG eliminates hallucination. It reduces it, but models still add unsupported details unless you constrain, cite, verify and measure.
Two kinds matter. Unfaithful (intrinsic): the answer contradicts or goes beyond the supplied context, measured by Faithfulness. Factual (extrinsic): the answer is wrong about the world, usually on long-tail facts from Parametric knowledge.
Kalai et al. (2025) argue the root cause is incentive: pretraining cannot separate true from merely plausible rare facts, and most benchmarks score 'I don't know' as zero, so models are trained to guess. In RAG systems, a large share of wrong answers start upstream: the right chunk was never retrieved, or was retrieved but ranked out of the context. Fix in layers: retrieval recall (Hybrid search, reranking), 'answer only from context' instructions, mandatory citations validated against chunk ids, calibrated Abstention & relevance thresholds, and claim-level faithfulness scoring with an LLM-as-judge on a Golden set plus sampled production traffic. Do Error analysis before choosing which layer to fix.
System prompt requires answers only from delimited context with a citation per claim; post-generation check that every cited id exists in the context; abstain when the reranker's top score is below a calibrated threshold; nightly claim-level faithfulness judge on 500 sampled answers; quarterly 200-answer Error analysis.
A telecom support bot's context says: 'International roaming is included in the Plus plan in EU countries.' The answer: 'Roaming is included in the Plus plan in the EU and the UK, up to 20 GB.' Claim-level faithfulness splits it into 3 claims (EU included, UK included, 20 GB cap); only 1 is supported, so the score is 0.33 and the answer is blocked and replaced with the supported sentence plus a citation.
Don't say 'use a better model'. Commit to a diagnosis: sample 200 wrong answers and label each as retrieval miss, ranking miss, unfaithful generation or bad source. Fix the largest bucket first, and measure retrieval and generation separately so each fix is attributable. The expected follow-up is 'how do you catch it in production?': claim-level faithfulness on a daily sample, citation-id validation on every response, and an abstention rate you track as a product metric.
Numbers worth knowing
- Faithfulness alert line
- < 0.90 claim supportjudge-scored daily sample of 500 answers on a grounded route
- Error-analysis sample
- 100–200 failuresenough to rank failure buckets with ±7-point confidence
- Abstention rate
- 2–8%healthy range on a customer-facing grounded route; 0% means the model is guessing
How to decide
| When | Do this |
|---|---|
| Error analysis shows most failures are retrieval misses | Invest in Recall@k: hybrid search, query rewriting, chunking; leave the prompt alone. |
| The right chunk is in context but the answer adds claims | Tighten instructions, require per-claim citations, and gate on faithfulness. |
| The route is regulated (health, legal, finance) or makes commitments | Block unsupported answers and route to a human; never ship ungrounded output. |
Cost maths
Faithfulness monitoring cost = sampled answers × judge tokens × price. Assume 500 answers/day × (4,000 in + 400 out) on a judge at $3 / $15 per M (illustrative): 500 × ($0.012 + $0.006) = $9/day, ≈ $270/month. Compare with one tribunal or refund-policy dispute: the monitoring is noise in the budget.
Operating it
- Faithfulness below 0.90 for two consecutive days
- Abstention rate falls toward zero after a prompt or model change
- Cited ids that do not exist in the context appear at all
- Claim-level faithfulness on a daily sample: alert below 0.90
- Invalid citation ids per 1k responses: alert on any
- Abstention rate: alert outside 1–10%
When it goes wrong
In the public Moffatt v. Air Canada case (BC Civil Resolution Tribunal, 2024), the airline's website chatbot told a customer he could claim a bereavement fare refund after travel; the actual policy did not allow it.
The customer was refused the refund and brought a claim with a screenshot of the chat.
The chatbot's answer contradicted the airline's own published bereavement-policy page and nothing checked it against that page. The airline argued the chatbot was responsible for its own statements; the tribunal rejected that and held the airline liable.
The general lesson: ground policy answers in the current policy text, cite it, block unsupported commitments, and treat bot output as the company's own statement.
Related terms
Where it shows up
Sources
- Survey of Hallucination in Natural Language Generation — Ji et al., 2022paper
- Why Language Models Hallucinate — Kalai et al., 2025paper
Check yourself
Three questions. Every option is explained once you answer.