What does an LLM actually compute when it answers a question?
- 1 min — the loop: logits → sample → append → repeat
- 2 min — prefill vs decode and the latency formula
- 2 min — consequences: hallucination, grounding, instability
- 2 min — how you'd instrument it in production (logprobs, stability evals)
One thing, repeatedly: a probability distribution over the next token given every token so far. The transformer maps the sequence to a vector of logits — one score per vocabulary entry (128,256 entries for Llama 3) — a sampler picks one token, it is appended, and the loop runs again until a stop token or max_tokens. That is next-token prediction. Reasoning, retrieval and planning are behaviours of that loop plus whatever sits in the context window.
answer = sample(model(prompt)) → append → repeat
Compute per generated token ≈ 2 × parameters FLOPs: ~16 GFLOPs for an 8B dense model, ~140 GFLOPs for 70B. The prompt is processed in one parallel pass (prefill); output is produced one token per forward pass (decode), each pass reading all the weights plus the KV cache.
A legal-contracts assistant gives a confident "the indemnity cap is 12 months of fees" on Monday and "24 months" on Tuesday for the same clause. Framed as sampling, the debug path is mechanical: (1) log top-5 logprobs on the first divergent token — here "12" had p=0.46 and "24" p=0.41, a near tie; (2) check whether the clause was even in context (it was truncated at 32k); (3) fix the retrieval so the clause is present, and the distribution collapses to p>0.95 on the right figure. Temperature was never the root cause.
- Model it as a conditional text generator, not a database. That predicts hallucination (fluent continuation with no fact behind it) and why grounding works (you change what it conditions on).
- Latency follows the loop: total ≈ TTFT + output_tokens × inter-token latency. At 40 ms/token, a 600-token answer is 24 s of decode however fast prefill is — cap output length before you buy GPUs.
- Answer stability is measurable: run 20 paraphrases × 5 samples on 200 high-stakes questions and alert if agreement drops below 90%.
- Org scope: the platform team owns the gateway that logs logprobs and token counts for every call; product teams own the stability evals for their domain. Buy the model, build the instrumentation — logprob logging is the one thing no vendor dashboard gives you per feature.
- Where it breaks: treating a single sample as the model's belief. Which fact surfaces depends on wording and on what else is in context.
70B dense model, BF16, single request: decode reads ~140 GB of weights per token. On 2×H100 (2 × 3.35 TB/s) that is ~21 ms/token at batch 1 ignoring overhead, so a 400-token answer is ≥8.4 s of decode. Prefill of 3,000 tokens ≈ 2 × 70e9 × 3,000 = 4.2e14 FLOPs; at ~800 TFLOPS effective across 2 GPUs ≈ 0.5 s. Decode dominates by ~16×.
What the answer sounds like at each level
Explains the next-token loop, sampling, and that the model has no lookup step — it continues text.
Adds prefill vs decode, why output length dominates latency, and uses logprobs to debug instability in a real feature.
Turns the mental model into org practice: logprob and token logging at the gateway, paraphrase-stability evals owned per domain, and output caps as the first latency lever.
Follow-ups the interviewer will push
1Why is generating 500 tokens slower than reading 5,000?
Prefill processes all 5,000 input tokens in one parallel, compute-bound pass; decode needs 500 sequential forward passes, each memory-bandwidth-bound on reading the weights and KV cache.
2Does the model 'know' it is uncertain?
Partially: token logprobs are a weak, uncalibrated signal. Use them for routing and debugging, and validate against labelled outcomes before using them for abstention decisions.
3Now the provider does not return logprobs. What do you do?
Use self-consistency: sample n=5 and measure agreement on the parsed answer. It costs 5× output tokens, so run it on a sampled 1–5% of traffic or only on high-stakes routes.
Red flags interviewers listen for
- Describes the model as searching a database or 'looking up' facts.
- Thinks temperature 0 makes answers correct rather than merely repeatable.
- Cannot say why output length drives latency more than input length.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | States next-token distribution, sampling, autoregressive append, stop conditions. |
| Quantitative | Gives FLOPs/token ≈ 2×params and the TTFT + n×ITL latency formula with a worked number. |
| Production judgement | Connects the model to debugging with logprobs and to measurable stability evals with thresholds. |
Sources
- Attention Is All You Need — Vaswani et al., 2017paper
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper