Faithfulness
Share of an answer's claims that the provided context supports. Measures grounding, not truth: a faithful answer from a wrong doc is still wrong.
Faithfulness checks whether the answer sticks to the sources it was given. You split the answer into small factual claims and check each one against the retrieved text. If 4 of 5 claims are supported, faithfulness is 0.8. A low score means the model added things that were not in the evidence, which is one kind of Hallucination.
A court reporter's transcript: it is faithful if every line matches what was said in the room, whether or not the witness was telling the truth.
Reading faithfulness as factual accuracy. If retrieval returned an outdated document, a perfectly faithful answer repeats the outdated fact.
Ragas-style computation: (1) an LLM decomposes the answer into atomic claims; (2) each claim is checked for entailment against the retrieved context by an LLM judge or NLI model; (3) faithfulness = supported / total claims. FActScore (Min et al. 2023) does the same against a knowledge source for long-form generation. Implementation choices that move the number: claim granularity (a sentence with three facts is one or three claims), whether 'partially supported' counts, and the judge model. Calibrate against 100–200 human-labelled claims and report agreement; LLM judges are lenient on paraphrased numbers and dates. Faithfulness is separate from answer correctness and from Context precision: you need all three to locate a failure. A pre-computed citation per claim (Attribution & citations) makes the check cheaper and more precise, since each claim is compared to one span instead of the whole context.
Ragas faithfulness or an in-house claim-split + judge with a pinned model; claims checked against cited spans when citations exist; 150 human-labelled claims for calibration; report the mean and the share of answers below 0.8; slice by claim type (numbers, dates, entities, negations).
Clinical intake summariser. Context: triage note and medication list. Summary claims: (1) 'chest pain onset 2 hours before arrival' ✓, (2) 'history of type 2 diabetes' ✓, (3) 'on metformin 500 mg BID' ✓, (4) 'no known drug allergies' ✗ (note says 'allergies: not assessed'), (5) 'troponin normal' ✗ (no lab results in context). Faithfulness = 3/5 = 0.6. Both failures are the model filling a default, which is the dangerous pattern in clinical text; the team adds a rule that missing fields are rendered as 'not documented'.
Say: faithfulness tells me whether generation respects the evidence, so I pair it with context recall for retrieval and answer correctness for end-to-end. I calibrate the judge on ~150 human-labelled claims and gate on the low tail, not the mean. Expect: 'faithfulness is 0.95, are we done?' Not if the 5% are clustered in numbers, dosages or dates; slice by claim type.
| Claim in summary | In context? | Verdict |
|---|---|---|
| Onset 2 h before arrival | Triage note | Supported |
| Type 2 diabetes | History | Supported |
| Metformin 500 mg BID | Med list | Supported |
| No known allergies | 'Not assessed' | Unsupported |
| Troponin normal | No labs given | Unsupported |
| Faithfulness | 3 / 5 = 0.60 |
Numbers worth knowing
- Production gate
- ≥ 0.9 mean, < 5% of answers below 0.8customer-facing RAG
- Judge-human agreement
- ≥ 85% on claim verdictsbefore trusting the metric for gating
- Cost per scored answer
- $0.002–0.01small judge, 3–8 claims, 3–5k tokens context
How to decide
| When | Do this |
|---|---|
| Regulated output (clinical, legal, financial) | Blocking faithfulness check; unsupported claims removed or answer withheld |
| High-volume consumer assistant | Sample 2–5% online, full golden set offline |
| Faithfulness low but context recall high | Prompt and decoding problem: tighten instructions, lower temperature, require citations |
Cost maths
Online sampled scoring: 1M answers/month × 3% × $0.005 = $150/month. Blocking on every answer: 1M × $0.005 = $5,000/month plus 300–800 ms latency; reserve that for regulated flows.
Operating it
- Unsupported-claim share in a high-risk slice (dosage, money) > 1%
- Judge drift: agreement on the calibration set falls below 85%
- New model or prompt version: full offline run before rollout
- Mean faithfulness on sampled traffic; alert < 0.88 (7-day)
- Share of answers with any unsupported numeric claim; alert > 1%
- Judge agreement on the frozen calibration set; re-check monthly, alert < 85%
When it goes wrong
Insurance summaries stated deductibles that did not match the policy schedule on about 4% of claims
Adjuster spot checks, then a faithfulness slice on numeric claims: 0.81 vs 0.97 overall
Policy schedules were tables; the parser flattened them, and the model picked a deductible from an adjacent row. The judge, checking text loosely, passed it
Layout-aware parsing for schedules, numeric claims checked by exact-value match against the cited span, and a numeric-claim slice in the gate
Related terms
Where it shows up
Sources
- Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023paper
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation — Min et al., 2023paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
Check yourself
Three questions. Every option is explained once you answer.