Next-token prediction
The single task an LLM is trained on and runs at inference: given the text so far, assign a probability to every possible next token.
At its core, an Large Language Model (LLM) does one thing: it looks at the text so far and predicts which Token (a word or piece of a word) should come next. It gives every possible token a probability, one is picked, it is added to the text, and the process repeats.
Long answers, code and step-by-step reasoning are all built this way, one token at a time.
Phone autocomplete, scaled up enormously: instead of suggesting the next word from your own messages, it has learned from a large slice of everything written.
Dismissing it as 'just autocomplete' and therefore incapable of reasoning, or the reverse, assuming fluent output means the model checked facts. The objective is simple; doing it well is not, and plausibility is not verification.
Training: for a sequence x₁..xₙ the model outputs p(x_t | x_<t) at every position in one parallel pass (the Transformer's causal mask), and the loss is the mean negative log-probability of the true next token. Perplexity is exp(loss). Inference is autoregressive: compute Logits & probabilities over the vocabulary (128,256 entries for Llama 3; ~200k for OpenAI's o200k tokenizer), apply decoding rules (Temperature & sampling, Top-p (nucleus) sampling, grammar masks), pick one token, append, repeat until a stop token or max_tokens.
Each output token is one full forward pass, which is why output tokens cost 4–5× input tokens on most price lists and why output length dominates latency. The objective rewards plausibility, not truth: fluent hallucinations follow directly from it. The model cannot revise emitted tokens; it can only generate more. Structured output / decoding works by masking invalid next tokens at each step, and reasoning models still do next-token prediction — they just emit thousands of tokens before the answer.
Always set max_tokens explicitly per route, always check the stop/finish reason ('length' or 'max_tokens' means truncated), stream output to users, and use Structured output / decoding for anything a program parses.
E-commerce catalog extraction. Prompt ends with '"color": "'. The model scores ' navy' 0.61, ' blue' 0.22, ' dark' 0.09 and spreads 0.08 over 128k other tokens. Greedy decoding takes 'navy', then predicts '"' at 0.97. A 400-token JSON answer is 400 sequential forward passes: at 18 ms per token that is 7.2 s of decode, versus ~0.2 s to prefill a 3,000-token product description.
Use it to explain failure modes, not to dismiss models. Say: 'The model is trained to be plausible, not correct; grounding, constrained decoding and evals close that gap.' Expect the follow-up 'then how can it reason?': predicting text well requires modelling structure, and RL post-training rewards correct final answers, so reasoning emerges inside the same loop.
| Candidate next token | Probability |
|---|---|
| navy | 0.61 |
| blue | 0.22 |
| dark | 0.09 |
| 128k others | 0.08 |
Numbers worth knowing
- Forward passes per answer
- 1 per output tokenplus one prefill for the prompt
- Output vs input price
- 4–5× per tokencommon on 2025–26 API price lists
- Llama 3 vocabulary
- 128,256 tokensone logit each per step
How to decide
| When | Do this |
|---|---|
| A program will parse the output | Constrain decoding with a JSON schema or grammar; never rely on prompt wording alone. |
| Answers must be factual about your data | Ground with Retrieval-Augmented Generation (RAG) and require citations; the objective alone gives plausibility. |
| Latency is too high | Cut output tokens first; each one is a sequential forward pass. |
Cost maths
Cost = input tokens × input price + output tokens × output price. Assume $3/M input, $15/M output (illustrative list prices, Oct 2026). 3,000-token prompt + 400-token JSON answer = $0.009 + $0.006 = $0.015. Trimming the answer to 150 tokens (drop echoed fields) saves $0.00375 per call — 25% of total cost at 1M calls/month = $3,750/month.
Operating it
- finish_reason = 'length' on > 0.5% of calls → raise max_tokens or shorten the schema
- Mean output tokens per call rises > 20% after a prompt change
- JSON parse failures > 0.1% → switch to constrained decoding
- Share of responses with finish_reason 'length' > 0.5%
- Output tokens per call p95 vs max_tokens — alert at > 80% of the cap
- Schema-validation failure rate > 0.1%
When it goes wrong
4% of invoice-extraction calls produced invalid JSON; the downstream job silently dropped those invoices for two weeks.
Finance reconciliation found missing invoices; logs showed finish_reason 'length' on every failed call.
max_tokens was 512, and invoices with 30+ line items needed ~700 output tokens; the code parsed the text without checking the finish reason.
Checked finish reason on every call, raised max_tokens to 2,048 for that route, switched to schema-constrained decoding, and sent parse failures to a dead-letter queue with an alert.
Related terms
Sources
- Language Models are Unsupervised Multitask Learners (GPT-2) — Radford et al., 2019paper
- Language Models are Few-Shot Learners — Brown et al., 2020paper
- Attention Is All You Need — Vaswani et al., 2017paper
Check yourself
Three questions. Every option is explained once you answer.