Temperature & sampling
A sampling setting that sharpens (low) or flattens (high) next-token probabilities, trading consistency for variety; it does not change what the model knows.
When an Large Language Model (LLM) picks its next Token, it has a probability for every option. Temperature controls how adventurous that pick is. Low temperature (like 0.2) makes it almost always pick the most likely word, so answers are consistent. High temperature (like 1.0) gives less likely words a real chance, so answers vary more.
Use low temperature for extraction, classification and code; use higher temperature for brainstorming and creative writing.
A restaurant regular. At low temperature they order their usual every time; at high temperature they sometimes try the specials, which is great for discovery and bad when you need predictability.
Assuming temperature 0 makes outputs perfectly reproducible, or that lowering temperature reduces hallucination. It reduces variance; a confidently wrong top answer stays wrong.
Sampling probabilities are softmax(zᵢ / T) over the Logits & probabilities z. T < 1 widens the gaps (sharper), T > 1 shrinks them (flatter), T → 0 is greedy decoding. Most stacks apply temperature first, then truncation (Top-p (nucleus) sampling, top-k, min-p).
T = 0 does not guarantee identical outputs: inference kernels are not batch-invariant, so the same request can produce different floating-point results depending on what else is in the batch. Thinking Machines (2025) sampled Qwen3-235B 1,000 times at temperature 0 and got 80 distinct completions. Reasoning models often fix temperature: OpenAI's o-series reject non-default values, and DeepSeek recommends 0.6 for R1 because greedy decoding causes repetition loops in long traces. Temperature changes how the model chooses, not what it knows: high values add low-probability errors; low values repeat the single best guess, including a wrong one.
Extraction, classification, SQL and code: 0–0.2. Chat assistants: 0.7. Ideation and marketing copy: 0.9–1.0 with top-p 0.95. Reasoning models: omit the parameter and use the provider default. Enforce per-model parameter allowlists in the LLM gateway.
E-commerce product descriptions. Logits for the next word after 'This jacket is': ' warm' 2.0, ' lightweight' 1.5, ' waterproof' 1.0, ' stylish' 0.5. At T = 0.2: 92% / 7.5% / 0.6% / 0.05%; at T = 1.0: 45% / 28% / 17% / 10%; at T = 1.5: 38% / 28% / 20% / 14%. For 50,000 catalog items, T = 0.2 yields near-identical openings on most items (bad for SEO), while T = 0.9 varies them; attribute extraction from the same items stays at T = 0.
Screening answer: 'temperature divides the logits before softmax; I use 0 for extraction and code, 0.7–1.0 for ideation, and I don't rely on 0 for reproducibility — I build evals that tolerate variance and pin seeds and model versions where supported.' Expect: 'does lower temperature reduce hallucination?' — it reduces variance, not error; a confidently wrong top token stays wrong.
Numbers worth knowing
- Distinct outputs at T = 0
- 80 of 1,000Qwen3-235B, Thinking Machines 2025, standard kernels
- DeepSeek-R1 recommended temperature
- 0.6avoids repetition loops in long traces
- Top-token share at T = 0.2
- ~92%logit gap of 0.5 to the runner-up
How to decide
| When | Do this |
|---|---|
| Output is parsed or compared to ground truth | T = 0–0.2 plus Structured output / decoding. |
| Output needs variety across many items | T = 0.8–1.0 with top-p 0.95, and dedupe downstream. |
| Model is a reasoning model | Do not send temperature; control behaviour with effort and prompts. |
Cost maths
Temperature itself is free; strategies built on it are not. Self-consistency with 5 samples at T = 0.7 costs 5× output tokens: 400 output tokens × 5 × $15/M = $0.03 per question vs $0.006 — justified only where the eval shows a gain.
Operating it
- Eval scores vary > 2 points between identical runs → report mean of 3 runs or use batch-invariant kernels
- Duplicate-output rate across items > 5% on a generation route → raise temperature or vary prompts
- Provider 400 errors on sampling parameters after a model change → gateway allowlists
- HTTP 400 rate per model > 0.5% after any model or parameter change
- Run-to-run variance of the golden-set score > 2 points
- Duplicate or near-duplicate output rate on generation routes > 5%
When it goes wrong
After a route was switched to a reasoning model, 100% of its requests failed with HTTP 400 for about 40 minutes.
Error-rate alert on the route; the error message named temperature as unsupported for the model.
The client always sent temperature = 0.2; the new model only accepts the default sampling settings.
Rolled back, then added per-model parameter allowlists in the gateway that strip unsupported parameters, plus a pre-deploy smoke test calling each new model with production parameters.
Related terms
Where it shows up
Sources
- The Curious Case of Neural Text Degeneration — Holtzman et al., 2019paper
- Defeating Nondeterminism in LLM Inference — He / Thinking Machines Lab, 2025blog
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI, 2025paper
- Reasoning models guide — OpenAI, 2026docs
Check yourself
Three questions. Every option is explained once you answer.