Foundations

Large Language Model (LLM)

Budding levelReviewed Oct 2026In 29 interview questionsIn 8 posts

A transformer trained to predict the next token over trillions of tokens, then tuned to follow instructions: the engine under chat, agents and RAG.

In plain English

A large language model is a program that has read an enormous amount of text and learned one skill extremely well: given some text, guess what comes next. It works one small piece at a time, where each piece is a Token (roughly a word fragment). Ask it a question and it continues your text with an answer, one token after another.

Chatbots, coding assistants and agents are all built by wrapping this next-token predictor with instructions, tools and data.

Analogy

An extraordinarily well-read autocomplete: it has seen so many sentences that its guesses about what comes next often look like reasoning.

Common pitfall

Treating the model as a database that looks facts up. It generates the most plausible continuation; without grounding it will hallucinate in the same confident tone it uses for correct answers.

How it actually works

A modern LLM is a decoder-only Transformer: stacked self-Attention and feed-forward (often Mixture of experts (MoE)) blocks. Pretraining on 10–40T tokens teaches Next-token prediction (Llama 3 used over 15T); post-training with Instruction tuning, RLHF (reinforcement learning from human feedback) or Preference optimization (DPO), and RL on verifiable rewards produces instruction following, tool use and reasoning.

Inference has two phases. Prefill runs the whole prompt through the network in one parallel pass and is compute-bound; it sets Time to first token (TTFT). Decode emits one token per step and is memory-bandwidth-bound: every step reads all weights plus the KV cache. An 8B model in BF16 is 16 GB, so on an H100 (3.35 TB/s) a single stream tops out near 200 tokens/s no matter how many FLOPs the chip has; Batching & throughput many streams is how servers recover throughput.

Each step produces Logits & probabilities over a 100k–250k vocabulary; a sampler (Temperature & sampling, Top-p (nucleus) sampling) picks one token and the loop repeats. Knowledge is frozen in the weights at the Knowledge cutoff, which is why production systems add retrieval, tools and Structured output / decoding.

What I'd ship on day one

Hosted frontier model behind an LLM gateway with Model fallback & routing to a second provider; streaming on; temperature 0 for extraction and classification, 0.7 for drafting; max_tokens set per route from measured p99 output + 30%; prompts in git with Prompt versioning; log input, output and reasoning tokens plus stop_reason on every call.

Worked example

A code-search assistant gets: 'Where do we retry failed S3 uploads?' The prompt is 6,000 tokens: 1,500 of instructions plus 8 retrieved code snippets and the question. On a hosted frontier model, prefill returns the first token in 300–600 ms; decode streams a 250-token answer at ~100 tokens/s, so the reply finishes in about 3 s. At an assumed $3 / M input and $15 / M output, the call costs $0.018 + $0.004 ≈ $0.022. Output tokens are 4% of the tokens but 17% of the bill.

In the interview

Default to a hosted frontier model behind a gateway for v1; move one narrow route to an 8–70B Open-weights vs closed models model on vLLM once it burns more than ~50M tokens/day and the Golden set gap is under 2 points. Expect the follow-up 'where does the latency go?': split it into TTFT (prefill, grows with prompt length) and inter-token latency (decode, grows with model size and batch), and say which one your product actually feels.

App

Tokenizer

GPU prefill

GPU decode

6,000-token promptApp → Tokenizer

6,000 ids, one passTokenizer → GPU prefill

KV cache + token 1GPU prefill → GPU decode

TTFT ≈ 300–600 msGPU prefill → App

1 token/step, ~10 msGPU decode → GPU decode

stream 250 tokensGPU decode → App

Prefill reads the whole prompt in parallel; decode loops one token at a time, re-reading weights and KV cache every step.

6k prompt, 250 out

2,950 ms

60k prompt, 250 out

5,500 ms

Reasoning model

17,950 ms

Prefill (TTFT)Visible outputHidden reasoning

Numbers worth knowing

Decode speed, hosted API: 50–150 tokens/s per streamsingle request on frontier models, Oct 2026; reasoning models spend hidden tokens before the first visible one
Weights memory: 2 bytes × parametersBF16: 8B ≈ 16 GB, 70B ≈ 140 GB; FP8 halves it, 4-bit quarters it
Single-stream decode ceiling: ≈ HBM bandwidth ÷ weight bytes8B BF16 on one H100 (3.35 TB/s) ≈ 200 tokens/s upper bound
Output vs input price: 4–5× per tokentypical hosted list prices, e.g. $3 in / $15 out per M (illustrative)

How to decide

When	Do this
Open-ended drafting or reasoning, under ~10M tokens/day	Use a hosted frontier model; self-hosting will not pay back the GPU and on-call cost.
Narrow high-volume task (classification, extraction, rewriting) and a small model is within 2 points on the golden set	Move it to a smaller hosted model or distil onto an 8B open-weights model on vLLM.
Answers must be current, private, permissioned or cited	Add Retrieval-Augmented Generation (RAG); never rely on the weights for those facts.
Users wait on long outputs	Stream, cap `max_tokens`, and pick the model by inter-token latency, not benchmark rank.

Cost maths

Cost per call = input_tokens × p_in + (output + reasoning tokens) × p_out. Assume $3 / M input and $15 / M output (illustrative list prices, Oct 2026). 6,000 in + 250 out = $0.018 + $0.00375 ≈ $0.022. At 200k calls/day that is ≈ $4,400/day, ≈ $132k/month. Switch the same route to a reasoning model that emits 2,000 hidden tokens and each call adds 2,000 × $15 / M = $0.03, more than doubling the bill. Prompt caching on the 1,500-token instruction prefix and the provider's batch API (~50% off) are the first two levers.

Operating it

Time to scale up when…

Spend on one route passes ~$20k/month and a smaller model is within 2 points on its golden set
TTFT p95 breaches the SLO because prompts routinely exceed 30k tokens
Provider 429 or 5xx rate above 1% of requests for a week

What to monitor

TTFT p95 per route: alert above 2× the 7-day baseline
stop_reason = max_tokens rate: alert above 0.5%
Cost per 1k requests per route: alert on +25% week over week
Provider error rate (429/5xx): alert above 1% over 15 minutes

When it goes wrong

What users saw

A contract-review assistant returned truncated JSON on ~4% of long master services agreements the week after a model version upgrade.

How it was caught

Parse-failure alarm on the extraction route, then a log query showing every failure had stop_reason = max_tokens.

Root cause

The new model version wrote longer clause summaries; max_tokens stayed at 1,024, so outputs were cut mid-object.

Fix

Set max_tokens from measured p99 output + 30% per route, moved extraction to Structured output / decoding, and added stop_reason distribution to the model-upgrade Eval gate (CI).

Where it shows up

BLOGBreaking into AI engineering: a roadmap from zero BLOGWhat happens when you ask an AI a question BLOGRAG in plain English: letting the AI look it up BLOGWhat is an AI agent, really?BLOGWorkflows before agents: pick the simplest pattern that works BLOGHow do you know if an AI is any good?BLOGThinking models, explained without the hype BLOGA reading list by level: the papers, posts and books worth your time I·1What does an LLM actually compute when it answers a question?I·4Why do LLMs hallucinate, and what actually reduces it?I·9How do you get reliable structured output from an LLM?I·8How do you build a cost model for an LLM feature before launch?II·1Why use RAG instead of fine-tuning an LLM?II·6How does query understanding affect retrieval quality?II·7What causes an LLM to ignore retrieved context?II·8How would you handle questions that cannot be answered from the knowledge base?II·1How would you choose the optimal chunk size?II·3How would you chunk tables, PDFs, and structured documents?II·5How would you handle documents containing images and text together?II·8How would you detect that your embedding model is the bottleneck?II·3When should you use a reranker after vector search?II·6How would metadata filtering improve retrieval?II·9How would you choose the value of top-k?II·10How would you design retrieval for queries containing multiple intents?II·2How would you prevent one customer's documents from appearing in another customer's results?II·8How would you handle access-controlled documents in RAG?II·10How would you design a RAG system supporting millions of documents and high query traffic?III·1What actually makes a system an agent, rather than an LLM call with extra steps?IV·1You have a new LLM feature and zero eval data. How do you build a golden set from scratch?IV·3How do you write a good rubric for grading open-ended LLM output?IV·8When is synthetic data useful for evals, and what are its risks?IV·4How do you put eval gates into CI for an LLM application?IV·5How do you A/B test an LLM change in production?IV·6How do you measure and control cost per query for an LLM product?IV·8How do you approach red teaming and jailbreak defence for an LLM product?IV·9How do you handle PII in an LLM system, from prompt to logs?IV·10How do you monitor quality drift for an LLM feature in production?

Sources

Attention Is All You Need — Vaswani et al., 2017paper
Training language models to follow instructions with human feedback — Ouyang et al., 2022paper
Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023paper

Check yourself

Three questions. Every option is explained once you answer.

1ConceptAn 8B model in BF16 serves a single stream on one H100 and tops out near 200 tokens/s even though the GPU has FLOPs to spare. Why?

2ScenarioA narrow classification route burns 60M tokens/day on a hosted frontier model. An 8B open-weights model scores 1.5 points lower on its golden set. What does the page recommend?

3CalculationA route sends 6,000 input tokens and gets 250 visible output tokens at $3/M input and $15/M output. You switch it to a reasoning model that also emits 2,000 hidden reasoning tokens. Roughly what does one call cost now?

Was this helpful?

Large Language Model (LLM)

Numbers worth knowing

#How to decide

#Cost maths

#Operating it

#When it goes wrong

Related terms