Logits & probabilities
The raw scores a model outputs for every token in its vocabulary; softmax turns them into next-token probabilities.
Before an Large Language Model (LLM) picks its next Token, it gives every token in its vocabulary (often 100,000 or more) a raw score called a logit. Higher means more likely. A step called softmax turns those scores into probabilities that add up to 100%.
Settings like Temperature & sampling and Top-p (nucleus) sampling work by adjusting logits or probabilities before a token is chosen.
Raw judge scores in a competition before they are converted into each contestant's percentage chance of winning.
Treating a high token probability as 'the model is 95% sure this fact is true'. It measures how likely that text is to follow, not whether it is correct; Calibration must be measured, not assumed.
The last hidden state is multiplied by the unembedding matrix to produce a vector z of vocabulary size V (128,256 for Llama 3). Probabilities are softmax(z)ᵢ = exp(zᵢ) / Σⱼ exp(zⱼ); only differences between logits matter. Log-probabilities are what APIs expose as logprobs (OpenAI Chat Completions returns up to 20 top alternatives per position; several providers, including Anthropic, do not expose them).
Uses: logit bias to ban or boost tokens; Structured output / decoding by setting invalid tokens to −∞ at each step; classification by comparing label-token logprobs; uncertainty signals (low top probability, high entropy); Model distillation from a teacher's full distribution. Caveats: token probabilities are not calibrated statements of factual confidence, especially after RLHF (reinforcement learning from human feedback) (Kadavath et al., 2022 found pretrained models reasonably calibrated on multiple choice, less so after RLHF); a multi-token answer's probability is the product across tokens and depends on tokenisation; and calibration shifts with every model version, so thresholds must be re-fit (Guo et al., 2017).
Use logprobs only where the provider exposes them; fit any confidence threshold on ≥ 1,000 labelled held-out items; track expected calibration error (ECE) per model version; prefer label tokens that are single tokens in the model's tokenizer.
Customer-support intent routing with label tokens ' billing' and ' technical': logprobs −0.11 and −2.30, i.e. 0.90 and 0.10. The team auto-routes when the top label ≥ 0.80 and sends the rest to a human: 6% of traffic. On 2,000 audited tickets, auto-routed accuracy is 97.5% and the human queue catches most of the hard cases.
Show you know softmax and logprobs, and that they are useful but uncalibrated: 'I use logprobs as a routing signal, fit the threshold on held-out data, and re-fit it on every model upgrade.' Expect: 'what if the API has no logprobs?' — use a verbalised confidence or a separate classifier, and calibrate either one the same way.
Numbers worth knowing
- Vocabulary (logits per step)
- 128,256Llama 3 tokenizer
- Top alternatives returned
- up to 20OpenAI Chat Completions top_logprobs
- Routing threshold example
- 0.80 → 6% to humanssupport intent routing, this page's example
How to decide
| When | Do this |
|---|---|
| You need a confidence signal and logprobs are available | Use top-label logprob with a threshold fit on held-out data. |
| Logprobs are not exposed | Train a small classifier or use LLM-as-judge confidence, and calibrate it. |
| The model version changes | Re-fit thresholds and re-check ECE before routing traffic. |
Cost maths
Logprobs are free tokens-wise on most APIs, but human review is not: at 6% routed to humans × 100k tickets/day × 2 minutes × $30/hour ≈ $6,000/day. Lowering the threshold from 0.80 to 0.70 might cut review to 3% ($3,000/day) — only if audited accuracy on the newly auto-routed slice stays above target.
Operating it
- ECE > 0.05 on the latest model version
- Human-review share moves > 50% after a model upgrade
- Auto-routed error rate > target on weekly audit
- Share of traffic below the confidence threshold — alert on ±50% change
- Expected calibration error per model version > 0.05
- Weekly audited accuracy of auto-handled items < target
When it goes wrong
After a model upgrade, the human-review queue shrank from 6% to 1% of tickets, and misrouted tickets in the auto path tripled.
Weekly audit of 500 auto-routed tickets showed accuracy falling from 97.5% to 92%.
The new model was more confident across the board; the 0.80 threshold fitted on the old model no longer meant the same thing.
Re-fit the threshold on 2,000 labelled tickets for the new version, added ECE and review-share checks to the upgrade checklist, and kept the old model on a Canary & ramped rollout until the audit passed.
Related terms
Sources
- On Calibration of Modern Neural Networks — Guo et al., 2017paper
- Language Models (Mostly) Know What They Know — Kadavath et al., 2022paper
- Attention Is All You Need — Vaswani et al., 2017paper
Check yourself
Three questions. Every option is explained once you answer.