Grounding & groundedness checks
Making the model answer from supplied sources (retrieved docs, tool results) rather than memory, and checking that it did.
A grounded answer is one built from evidence you gave the model, such as retrieved documents or a database lookup, instead of from what it half-remembers from training (Parametric knowledge). You ask for it in the instructions, you give the model good evidence, and then you check the answer against that evidence. Grounding is the main defence against Hallucination in Retrieval-Augmented Generation (RAG) systems.
An open-book exam where the rule is 'only answer from the textbook, and give page numbers'.
Believing an instruction like 'only use the context' grounds the model. Without a measured faithfulness check you have a hope, not a guarantee; models still blend in training knowledge, especially when the context is thin.
Grounding has three parts and all three must exist. Supply: retrieval or tools put the right evidence in context; if recall is bad, nothing else saves you. Instruct: the system prompt restricts the model to the sources, requires citations, and defines what to do when evidence is missing (see Abstention & relevance thresholds). Verify: a post-check measures Faithfulness, usually an NLI model or an LLM-as-judge that splits the answer into claims and checks each against cited chunks; unsupported claims are removed, flagged, or trigger a retry. Native citation features (Anthropic Citations, OpenAI file search annotations) return character spans and make the verify step cheaper. Grounding does not make sources correct: a grounded answer from a stale document is still wrong, so freshness and Chunk metadata matter as much as the prompt.
System prompt: answer only from tagged sources, cite ids for every factual sentence, say you cannot find it when sources are insufficient. Use the provider's native citations if available. Post-check with an LLM judge or NLI model; block answers with faithfulness < 0.8 or any uncited numeric claim. Show sources in the UI.
Insurance claims assistant: 'Is water damage from a burst pipe covered under policy HO-3 for claim 88213?'. Retrieval returns the policy's Section I perils clause and the claim's adjuster notes. The answer cites both: 'Covered as sudden and accidental discharge [S2 §I.B.12]; gradual seepage over 14+ days is excluded [S2 §I.2.c]; adjuster notes say the leak was found same day [S4].' The verifier checks each of the three claims; all are entailed, so it ships. When the question was about a policy whose text was not indexed, the bot replied that it could not find the policy instead of quoting a generic HO-3.
Say: grounding is supply, instruct, verify; most teams do the middle one only. I show the verify step with a faithfulness metric and a threshold that blocks or flags, and I separate 'retrieval missed' from 'model ignored evidence' using an oracle run. Follow-up to expect: 'how much does the verifier cost?' Roughly one small-model call per answer, 10–20% of the main generation cost.
Numbers worth knowing
- Verifier cost
- 10–20% of generation costsmall judge model checking 3–8 claims per answer
- Faithfulness gate
- ≥ 0.8–0.9 supported-claim ratioregulated or customer-facing answers; tune on human-labelled sample
- Verifier added latency
- 300–800 mssmall model, run after streaming completes or on the final answer
How to decide
| When | Do this |
|---|---|
| Customer-facing answers with legal or financial consequence | Block-on-fail verifier plus visible citations |
| Internal productivity tool | Async verifier that samples 5–10% and feeds error analysis |
| Context is empty or below a retrieval-score floor | Abstain before calling the LLM |
Cost maths
Main answer: 5k input + 400 output at $3/$15 per M = $0.021. Verifier on a small model at $0.80/$4 per M (illustrative, Oct 2026): 5k in + 150 out ≈ $0.0046, ~22% overhead. Sampling 10% instead of 100% cuts it to ~2%.
Operating it
- Unsupported-claim rate > 5% on sampled traffic: move verifier from sampled to blocking
- Verifier latency pushes p95 past the SLO: run it on a smaller model or after streaming
- Regulated tenant onboarded: blocking verifier and audit log required
- Faithfulness score on sampled answers; alert if the 7-day mean drops below 0.85
- Share of answers with zero citations; alert > 3%
- Retrieval-empty rate (no chunk above score floor); alert on a > 50% rise
When it goes wrong
An airline's website chatbot told a customer he could claim a bereavement fare refund retroactively; the policy said otherwise
The customer was refused, sued, and in Moffatt v. Air Canada (BC Civil Resolution Tribunal, 2024) the airline was held liable for its chatbot's statement
The answer was not tied to the actual policy text and nothing checked it against the published policy
General lesson: ground answers in the current policy, cite it, verify claims against it, and abstain when the policy is not retrieved; the company is responsible for what its bot says
Related terms
Where it shows up
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper
- Citations — Anthropic, 2025docs
- Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.