Top-p (nucleus) sampling
Nucleus sampling: sample only from the smallest set of tokens whose probabilities add up to p (e.g. 0.9), cutting off the unlikely tail.
Top-p is another way to control how an Large Language Model (LLM) picks its next Token. Instead of considering every possible word, it keeps only the most likely ones until their probabilities add up to p, for example 90%, and picks from that shortlist.
This stops the model from occasionally choosing a bizarre, very unlikely word, while still allowing variety among sensible options. It is used together with Temperature & sampling.
Choosing a restaurant from the places that together get 90% of the town's visits, ignoring the hundred obscure spots that almost nobody goes to.
Tuning temperature and top-p together by trial and error. The effects interact; fix one (top-p at 0.9–1.0), tune the other, and measure on a fixed eval set.
Sort tokens by probability, keep the smallest prefix whose cumulative probability ≥ p (the nucleus), renormalise and sample within it (Holtzman et al., 2019). Unlike top-k (Fan et al., 2018), the nucleus adapts: when the model is confident it is one or two tokens; when uncertain, hundreds. Holtzman et al. showed pure sampling from the full distribution produces incoherent text because of the long tail, while greedy decoding produces repetition; the nucleus avoids both.
Order matters: most stacks apply temperature, then top-p. p = 0.9–0.95 is standard for open-ended generation; p = 1.0 disables it. Tune one knob at a time — changing temperature and top-p together makes behaviour hard to reason about. Min-p (keep tokens with probability ≥ min_p × top probability, e.g. 0.05–0.1) behaves better at high temperature and is supported by vLLM, SGLang and llama.cpp. Many reasoning models restrict top-p as well as temperature (OpenAI's o-series reject non-default values). Like temperature, top-p controls variety, not factuality.
top_p = 1.0 for extraction and code at low temperature; 0.95 for open-ended generation; min-p 0.05 on self-hosted stacks when running temperature > 1. Never change both knobs in one experiment; omit both for reasoning models.
Developer-docs assistant suggesting code identifiers. Next-token probabilities: ' getUser' 0.47, ' fetchUser' 0.29, ' loadUser' 0.17, ' getUsr' 0.05, 128k others 0.02. With p = 0.9 the nucleus is {getUser, fetchUser, loadUser} (cumulative 0.93); the typo 'getUsr' can never be sampled. With p = 0.5 only getUser and fetchUser survive. With p = 1.0 at T = 1.2, the typo is sampled ~6% of the time.
Contrast with top-k (fixed count vs adaptive probability mass) and min-p (relative threshold). Recommend: 'fix top-p at 0.95 or 1.0, tune temperature, and measure on a fixed eval.' Expect: 'why not both low?' — very low top-p with low temperature collapses to near-greedy and invites repetition loops.
| Token | Probability | Cumulative | In nucleus (p = 0.9)? |
|---|---|---|---|
| getUser | 0.47 | 0.47 | yes |
| fetchUser | 0.29 | 0.76 | yes |
| loadUser | 0.17 | 0.93 | yes (crosses 0.9) |
| getUsr | 0.05 | 0.98 | no |
| 128k others | 0.02 | 1.00 | no |
Numbers worth knowing
- Standard nucleus
- p = 0.9–0.95open-ended generation
- Min-p threshold
- 0.05–0.1relative to the top token's probability
- Nucleus size
- 1 to hundreds of tokensadapts to model confidence per step
How to decide
| When | Do this |
|---|---|
| Rare garbage tokens appear in creative output | Lower top-p to 0.9 or add min-p 0.05 before lowering temperature. |
| Outputs repetitive or loop | Raise top-p to 0.95–1.0 and check temperature is not near 0. |
| Provider is a reasoning model | Do not send top_p. |
Cost maths
No direct cost. Indirect: degenerate repetition from over-truncation runs to max_tokens. If 2% of 1M calls loop to a 4,096-token cap instead of a 300-token answer, that is 20k × 3,796 extra tokens × $15/M ≈ $1,140/month of waste plus timeouts.
Operating it
- Repetition (same 4-gram ≥ 3 times) in > 1% of outputs
- Responses hitting max_tokens > 0.5% on a short-answer route
- Sampling config differs between environments (config drift)
- Repetition rate (repeated 4-grams) > 1% of outputs
- Responses ending at max_tokens > 0.5%
- Sampling parameters per route differ from the versioned config → alert
When it goes wrong
A catalog description generator started producing near-identical, looping descriptions; some ran to the token limit repeating one phrase.
Duplicate-description rate jumped from 1% to 34% on the catalog QA dashboard, and output tokens per call tripled.
A debugging change left top_p = 0.1 in the shared config with temperature 0.3, collapsing sampling to near-greedy.
Restored top_p = 0.95, moved sampling parameters into versioned per-route config with review, and added repetition and max-token-hit alerts.
Related terms
Sources
- The Curious Case of Neural Text Degeneration — Holtzman et al., 2019paper
- Hierarchical Neural Story Generation (top-k sampling) — Fan et al., 2018paper
- Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs — Nguyen et al., 2024paper
Check yourself
Three questions. Every option is explained once you answer.