How do you choose temperature and other sampling settings?
- 1 min — what T, top-p, top-k do to the distribution
- 2 min — per-task settings with a concrete product
- 2 min — non-determinism and reasoning-model constraints
- 2 min — testing and gateway policy
Sampling turns logits into a choice. Temperature divides logits before softmax: T<1 sharpens toward the top token, T>1 flattens. Top-p (nucleus) keeps the smallest set of tokens whose probabilities sum to p; top-k keeps k tokens.
Two facts candidates get wrong: (1) T=0 is not deterministic on hosted APIs. Requests are batched with other traffic, and GPU kernels whose reduction order depends on batch size produce slightly different floating-point results; MoE routing adds more variance. A near-tie token flips and the continuation diverges. Bit-exact output needs batch-invariant kernels on infrastructure you control. (2) Many reasoning models fix sampling: OpenAI o-series models reject non-default temperature, and Anthropic extended thinking is incompatible with temperature/top-k changes (as of Oct 2026; check current docs).
An insurance-claims product has three LLM steps. Extraction of policy number, loss date and amount into a schema: T=0, because consistency helps auditing. Drafting the adjuster's customer letter: T=0.7, generate 3 variants, adjuster picks one. The routing agent's tool-call step: T=0.2. Tests assert on parsed fields and rubric scores, never exact strings, because even the T=0 step differs on ~1–3% of reruns in the team's 1,000-item replay (illustrative measurement).
- Set sampling per step, version it with the prompt, and hold it fixed in the eval harness — changing T changes your eval numbers.
- Low temperature buys repeatability, not correctness. A wrong answer at T=0 is wrong every time.
- Need diversity with quality? Sample n candidates at T≈0.7–1.0 and select with a verifier or rubric — best-of-n beats a single high-T sample.
- Org scope: the gateway enforces per-route sampling defaults and strips parameters a model rejects, so 30 product teams don't each rediscover that a reasoning model 400s on
temperature=0. - Where it breaks: caching or regression tests keyed on exact output strings. Key caches on inputs; test on structure.
What the answer sounds like at each level
Knows low T for extraction, higher T for creative work, and what top-p does.
Explains why T=0 is not deterministic on hosted APIs and designs tests on parsed fields; knows reasoning models fix temperature.
Encodes per-route sampling policy in the gateway, uses best-of-n with verifiers where diversity matters, and treats sampling config as versioned prompt state.
Follow-ups the interviewer will push
1You need byte-identical outputs for an audit. How?
Self-host with batch-invariant kernels and fixed seeds, or store the generated output as the record of truth and never regenerate. Hosted APIs do not guarantee it.
2Temperature 1.5 to get 'more creative' copy?
Above ~1.2 the tail tokens get enough mass to produce incoherent text. Keep T≤1.0, add top-p 0.9–0.95, and get variety from n samples and prompt variation.
Red flags interviewers listen for
- Believes T=0 guarantees identical output on a hosted API.
- Uses one global temperature for every step in an app.
- Writes snapshot tests that compare exact model strings.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | Correctly describes logit scaling and nucleus truncation. |
| Reality check | Explains batch-dependent floating-point non-determinism and fixed-temperature reasoning models. |
| Engineering | Per-step settings, structural tests, best-of-n with verification. |
Sources
- The Curious Case of Neural Text Degeneration — Holtzman et al., 2019paper
- Defeating Nondeterminism in LLM Inference — Thinking Machines Lab (He et al.), 2025blog