Calibration
Whether stated confidence matches actual accuracy: of answers given 80% confidence, about 80% should be right. Needed for abstention and routing thresholds.
A model is well calibrated when its confidence means something: if it says it is 90% sure on 100 answers, about 90 of them should be correct. Poorly calibrated models are confidently wrong, or needlessly unsure.
Calibration matters whenever you act on a confidence score: deciding when the system should say 'I don't know' (Abstention & relevance thresholds), when to escalate to a human, or how far to trust an LLM-as-judge verdict.
A weather forecaster whose '70% chance of rain' days really do see rain about 70% of the time; you can plan around that forecaster, and not around one who says 70% every day.
Asking the model 'how confident are you from 0 to 100?' and thresholding on the answer. Instruction-tuned models say 85–95 almost regardless of correctness, so the threshold filters nothing.
Measure with a reliability diagram (bin predictions by confidence, plot accuracy per bin) and Expected Calibration Error: ECE = Σ_bins (n_b / N)·|acc_b − conf_b|. With 10 bins you need roughly 1,000 labelled predictions for a stable estimate. Guo et al. (2017) showed modern neural networks are overconfident and that temperature scaling, one parameter fitted on held-out data, fixes much of it. Kadavath et al. (2022) found large language models reasonably calibrated on well-formatted multiple-choice questions when you read token probabilities from the Logits & probabilities, and that they can be trained to predict P(True) for their own answers.
In production you usually lack clean probabilities: verbalised confidence ('I am 95% sure') from instruction-tuned models clusters at 80–95% regardless of correctness, and hosted APIs may not expose log-probs. Practical signals: agreement across 3–5 samples (self-consistency), a trained verifier or Reranker score, retrieval score of the best supporting passage, or log-probs of a constrained yes/no judgement. Whatever the signal, fit a mapping (isotonic regression or Platt scaling) on 500–1,000 labelled examples from your own traffic, then choose thresholds from the coverage–accuracy curve, and recalibrate after every model or prompt change, because calibration does not transfer across versions.
Confidence signal = agreement across 3–5 samples and/or a verifier score (not verbalised confidence); isotonic regression (sklearn IsotonicRegression) fitted on 500–1,000 labelled production examples; reliability diagram and ECE with 10 bins in the eval report; thresholds chosen from the coverage–accuracy curve and the cost ratio of errors to reviews; recalibration in the eval gate on every model or prompt change.
Insurance-claims triage: auto-approve when confidence is high, else route to an adjuster. Raw verbalised confidence: 92% of claims at '≥90%', of which only 81% were correct. Using agreement across 5 samples plus a verifier score, isotonic-calibrated on 1,200 labelled claims: ECE falls from 0.11 to 0.03. Threshold 0.95 auto-approves 58% of claims at 98.6% accuracy (95% CI ±0.9 at n≈700 in the held-out set); threshold 0.90 would approve 71% at 96.2%. The business picks 0.95 because a wrong auto-approval costs 40× an adjuster review.
Say: "I never threshold on raw verbalised confidence; I calibrate a signal like sample agreement or a verifier score on our own labels, check ECE and the coverage–accuracy curve, and recalibrate after every model change." Expect: "how do you pick the threshold?": from the cost of a wrong automatic action vs a human review, read off the curve.
Numbers worth knowing
- Labelled examples for a stable 10-bin ECE
- ~1,000 (500 minimum)fewer leaves bins with too few points
- Usable ECE target
- ≤0.05for thresholds that trigger automatic actions
- Self-consistency samples
- 3–5 per querymultiplies generation cost by the same factor
How to decide
| When | Do this |
|---|---|
| A confidence score triggers an automatic action (approve, send, close) | Calibrate on your own labels and choose the threshold from the cost of a wrong action vs a review. |
| The API exposes log-probs and the decision is a constrained choice | Use token probabilities of the choice, temperature-scaled on held-out data. |
| Only free text is available | Use sample agreement or a separate verifier, never the model's stated percentage. |
Cost maths
Self-consistency with 5 samples multiplies generation cost 5×: a $0.01 classification becomes $0.05. On 100k claims/month that is $5k vs $1k, which buys auto-approval of 58% of claims; if an adjuster review costs $4, each auto-approved claim saves ~$4 and the 58k auto-approvals save ~$232k against $4k extra model cost. A verifier model at $0.002 per call is a cheaper signal when it calibrates as well.
Operating it
- Automatic actions exceed ~1,000 a day: monitor calibration weekly on a labelled audit sample.
- More than one model or prompt version in production: calibrate each separately; mappings don't transfer.
- ECE on a weekly labelled audit sample: alert above 0.05.
- Accuracy of automatic actions above the threshold: alert when the CI lower bound falls below target.
- Share of traffic above threshold: alert on a ±10-point shift (the score distribution moved).
When it goes wrong
After a model upgrade, an auto-close rule for low-risk support tickets ('confidence ≥0.9') started closing tickets that customers re-opened at 3× the previous rate.
Re-open rate alert on auto-closed tickets, two weeks after the upgrade.
The isotonic mapping had been fitted to the previous model's verifier scores; the new model shifted the score distribution upward, so 'calibrated 0.9' now corresponded to about 0.78 accuracy.
Made recalibration a required step in the model-migration checklist, added a weekly 200-ticket labelled audit with an ECE alert, and paused automation automatically when ECE exceeds 0.05.
Related terms
Where it shows up
Sources
- On Calibration of Modern Neural Networks — Guo et al., 2017paper
- Language Models (Mostly) Know What They Know — Kadavath et al., 2022paper
Check yourself
Three questions. Every option is explained once you answer.