The three dials: tokens, temperature and context

The first bug: a nightly job summarises support tickets, and a week later someone notices that about one summary in twenty-five ends mid-sentence. The output cap was 256 tokens, the long tickets needed more, and nobody checked the response's stop reason, so truncated text flowed into the database looking exactly like finished text.

The second bug: an extraction test is flaky, so the team sets temperature to 0 to make it “deterministic”. It still fails one run in fifty with a slightly different output. Nothing is wrong with the test. Temperature 0 on a hosted API does not mean what they assumed.

Both bugs live in the three settings every request carries: tokens, temperature and the context window. Each dial below comes with code that runs on the Python standard library.

Dial one: tokens

Length limits, prices and generation time are all measured in tokens. For napkin maths, English prose averages about 4 characters per token. Use that heuristic only for napkins. Code, numbers, JSON and non-English text run denser, and two tokenizers can disagree by 20% or more on the same paragraph, so a budget computed for one model is wrong for another.

For a real count, use the target model's own tokenizer:

OpenAI models: the tiktoken library, with o200k_base for GPT-4o and later models (cl100k_base for GPT-4 and GPT-3.5).
Claude: the tokenizer is not published; call the Messages API's count_tokens endpoint, which is free and accepts the same payload as a real request.
Open-weights models such as Llama, Qwen or Mistral: AutoTokenizer.from_pretrained(<model id>) from Hugging Face, which loads the exact vocabulary the model was trained with.

Dial two: temperature

At every step the model emits a raw score, a logit, for every token in its vocabulary. Softmax turns the scores into probabilities, and temperature divides every logit before softmax. A small divisor stretches the gaps, so the favourite dominates; a large divisor shrinks them, so the tail gets a chance. T = 0 cannot be computed that way, since dividing by zero is undefined. Every serving stack special-cases it as greedy decoding: take the highest-scoring token. The function below does the same.

example.pypython32 lines

import math
import random


def softmax_with_temperature(logits: list[float], temperature: float) -> list[float]:
    """Probabilities after temperature scaling. T == 0 means greedy: all mass on the argmax."""
    if temperature < 0:
        raise ValueError("temperature must be >= 0")
    if temperature == 0:
        best = max(range(len(logits)), key=lambda i: logits[i])
        return [1.0 if i == best else 0.0 for i in range(len(logits))]
    scaled = [x / temperature for x in logits]
    biggest = max(scaled)  # subtract the max so exp() cannot overflow
    exps = [math.exp(x - biggest) for x in scaled]
    total = sum(exps)
    return [e / total for e in exps]


def main() -> None:
    words = ["Paris", "Lyon", "London", "banana"]
    logits = [6.0, 3.5, 3.0, 0.5]
    rng = random.Random(42)  # fixed seed so the run is repeatable
    for t in (0.0, 0.2, 1.0, 2.0):
        probs = softmax_with_temperature(logits, t)
        row = "  ".join(f"{w}={p:.2f}" for w, p in zip(words, probs))
        picks = rng.choices(words, weights=probs, k=10)
        print(f"T={t}: {row}")
        print(f"       10 samples: {' '.join(picks)}")


if __name__ == "__main__":
    main()

texttext8 lines

T=0.0: Paris=1.00  Lyon=0.00  London=0.00  banana=0.00
       10 samples: Paris Paris Paris Paris Paris Paris Paris Paris Paris Paris
T=0.2: Paris=1.00  Lyon=0.00  London=0.00  banana=0.00
       10 samples: Paris Paris Paris Paris Paris Paris Paris Paris Paris Paris
T=1.0: Paris=0.88  Lyon=0.07  London=0.04  banana=0.00
       10 samples: Paris Paris Paris Paris London Paris Paris Paris Paris Paris
T=2.0: Paris=0.64  Lyon=0.18  London=0.14  banana=0.04
       10 samples: Lyon Lyon Paris banana Paris Paris London Paris London Paris

Temperature never reorders the options; Paris is always the favourite. It changes how much the favourite dominates. At 2.0, “banana” appears once in ten samples from a 4% probability, and in a real answer that one token would steer every token after it.

Top-p cuts the tail instead of reshaping it: it keeps the smallest set of tokens whose probabilities sum to p and samples from those. Tune one of temperature or top-p and leave the other at its default; moving both makes any change impossible to attribute.

Reasoning models take the dial away. OpenAI's o-series rejects a custom temperature or top_p (only the default is accepted), and Anthropic's extended thinking does not allow changing temperature or top-k and limits top-p to 0.95–1. Code that hard-codes temperature=0 gets an API error when someone swaps in a reasoning model. Put consistency where it survives that swap: structured output, validation and an eval set.

Dial three: the context budget

The context window is one budget shared by instructions, history, retrieved text, the question and the answer. For a reasoning model the hidden thinking tokens come out of the same output allowance. Plan it in code, reserving the output first, so that you choose what to drop instead of letting an API error or silent truncation choose for you:

example.pypython32 lines

def rough_token_count(text: str) -> int:
    """Napkin estimate for English prose: ~4 characters per token.
    Replace with the target model's tokenizer before trusting the number."""
    return max(1, round(len(text) / 4))


def plan_context(parts: dict[str, str], window: int, reserve_output: int, reserve_thinking: int = 0) -> bool:
    """Reserve output (and thinking) tokens first; the input gets what is left."""
    budget = window - reserve_output - reserve_thinking
    used = 0
    for name, text in parts.items():
        n = rough_token_count(text)
        used += n
        print(f"{name:<13}{n:>7,} tokens")
    print(f"{'input total':<13}{used:>7,} of {budget:,} available after reserving {reserve_output + reserve_thinking:,}")
    return used <= budget


def main() -> None:
    parts = {
        "instructions": "Answer only from the API reference excerpts. Cite the endpoint name. " * 12,
        "history": "User asked how pagination works; assistant explained cursor tokens. " * 60,
        "excerpts": "GET /v2/orders returns 50 items per page; pass next_cursor to continue. " * 600,
        "question": "Why does my loop stop after page 3 of /v2/orders?",
    }
    print("fits:", plan_context(parts, window=16_000, reserve_output=1_000))
    print()
    print("fits:", plan_context(parts, window=16_000, reserve_output=1_000, reserve_thinking=4_000))


if __name__ == "__main__":
    main()

texttext13 lines

instructions     207 tokens
history        1,020 tokens
excerpts      10,800 tokens
question          12 tokens
input total   12,039 of 15,000 available after reserving 1,000
fits: True

instructions     207 tokens
history        1,020 tokens
excerpts      10,800 tokens
question          12 tokens
input total   12,039 of 11,000 available after reserving 5,000
fits: False

The same developer-docs request fits a standard model with 3,000 tokens to spare and fails by 1,039 once 4,000 thinking tokens are reserved. The retrieved excerpts are 90% of the input, so they are the thing to cut: rerank and send the best few excerpts rather than every match.

Large windows tempt you to skip this planning. Two costs argue against it. Every input token is processed before the first output token appears, so time to first token grows with the prompt. And models use the middle of a long input less reliably than its start and end, the lost-in-the-middle effect, so the fact you most need should sit near the top or bottom, not on page 40.

Starting settings

Task	Temperature	Context	Max output
Extract fields from a claim form	0 (greedy)	The one form, 1–3k tokens	300–500
Answer from developer docs	0–0.3	5–8 excerpts, ~4k tokens	400–800
Summarise a 60-page filing	0.3	20k-token parts, then merge	800–1,500
Brainstorm product names	0.9–1.0	A short brief, <1k tokens	200–400
Any of the above, reasoning model	Provider-fixed	Same as above	Add 2–8k for thinking

Day-one defaults, to be adjusted against your own eval set. Reasoning models ignore or reject the temperature column.

Key takeaways

Truncated output looks finished to every downstream system; the stop reason is the only reliable signal, so check it on every call.
Temperature 0 buys repeatability on your laptop, not on a shared GPU. Test parsed properties, not exact strings.
Token counts belong to a specific tokenizer; switching models silently invalidates every budget you computed for the old one.
Reserve output and thinking tokens first and give the input what is left; it is the input you can trim without losing the answer.

Sources

The Curious Case of Neural Text Degeneration — Holtzman et al., 2019paper
Defeating Nondeterminism in LLM Inference — Horace He, Thinking Machines Lab, 2025blog
Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023paper
tiktoken — OpenAI, 2022docs

#Dial one: tokens

#Dial two: temperature

#Dial three: the context budget

#Starting settings