LLM Foundations

A transformer trained to predict the next token over trillions of tokens, then tuned to follow instructions: the engine under chat, agents and RAG.

The unit an LLM reads and writes: a word or word fragment. Context limits, latency and price are all metered in tokens, not characters.

The maximum number of tokens, prompt plus output, a model can handle in one call. Anything outside it does not exist for the model.

The full input sent to an LLM: instructions, examples, retrieved context and the user's message. The main lever for controlling behaviour.

Facts an LLM absorbed into its weights during training: broad but frozen at the cutoff, unattributable, and impossible to update or delete one at a time.

When an LLM states something fluent and confident that is false or not supported by its sources.

Continuing to train a pretrained model on your own examples to change its behaviour, format or style. Not a reliable way to add facts.

Retrieval-augmented generation: fetch relevant documents for a question, put them in the prompt, and have the LLM answer from them with citations.

The full collection of documents a retrieval system can search: the source of truth that RAG answers are built from.

The transformer mechanism that lets each token weigh every earlier token in the context when building its representation.

Reusing the model's computed state for an identical prompt prefix across requests: cheaper, faster input for repeated context, minus a cache-write premium.

Constraining generation so output always matches a schema or grammar (e.g. valid JSON) by masking tokens that would break it.

Training a smaller, cheaper student model to imitate a larger teacher model's outputs on your task.

Models or pipelines that handle more than text, such as images, scanned PDFs, audio or video, in the same input or index.

Having an LLM translate a natural-language question into a SQL query, run it under a restricted role, and answer from the result.

RAG where the LLM decides what to retrieve, from which source, and when to search again, in a bounded loop instead of one fixed retrieval step.

Using an LLM with a rubric to grade outputs (faithfulness, relevance, correctness) so evaluation scales beyond human review.

A function built from layers of weighted sums and non-linearities whose weights are learned from examples by gradient descent; the machinery under every LLM.

The learned numbers (weights and biases) inside a model. '70B' means 70 billion of them; they set memory and compute cost, not quality on their own.

The architecture behind nearly all modern LLMs: stacked attention and feed-forward blocks that process every token of a sequence in parallel.

The first and most expensive training phase: a model learns language and world knowledge by predicting the next token across trillions of tokens.

The single task an LLM is trained on and runs at inference: given the text so far, assign a probability to every possible next token.

The text, code and examples a model learns from; mix, quality, recency and licensing decide what it knows and how it behaves.

Empirical power laws: loss falls predictably as parameters, data and compute grow, with diminishing returns per extra 10× of compute.

Supervised fine-tuning on (instruction, good response) pairs that turns a base model that continues text into an assistant that follows requests.

Reinforcement learning from human feedback: train a reward model on human preference comparisons, then optimise the LLM to score highly on it.

Methods like DPO that train directly on chosen-vs-rejected answer pairs, getting RLHF-like alignment without a reward model or RL loop.

Low-Rank Adaptation: fine-tune by training small add-on matrices while the original weights stay frozen; cheap, fast, swappable per task or tenant.

A model adapting to a task purely from instructions and examples in its prompt, with no change to its weights.

Including a handful of worked input-output examples in the prompt so the model copies the format, labels and style you want.

Having a model write intermediate reasoning steps before its final answer, which improves accuracy on multi-step problems at a token cost.

Models trained with RL to think at length before answering, trading latency and output tokens for accuracy on maths, code and multi-step work.

A sampling setting that sharpens (low) or flattens (high) next-token probabilities, trading consistency for variety; it does not change what the model knows.

Nucleus sampling: sample only from the smallest set of tokens whose probabilities add up to p (e.g. 0.9), cutting off the unlikely tail.

The raw scores a model outputs for every token in its vocabulary; softmax turns them into next-token probabilities.

Running a trained model to produce output. For LLMs: a compute-bound prefill over the prompt, then a memory-bound decode loop, one token per step.

Stored attention keys and values for past tokens, so each new token skips recomputing the sequence; at long context it fills GPU memory.

Storing weights (and sometimes activations or KV) in 8 or 4 bits instead of 16 to cut memory and speed up inference, at a quality cost you must measure.

Latency from sending a request until the first output token arrives; the main driver of how responsive a streaming LLM app feels.

Running many requests through the GPU together so one weight read serves many tokens: throughput and cost per token improve, per-stream speed falls.

A cheap draft proposes several tokens and the big model verifies them in one pass: same output distribution, fewer sequential big-model steps.

An architecture where each token is routed to a few of many expert sub-networks, so a huge model computes like a much smaller one per token.

Open-weights models publish their parameters for self-hosting; closed models are API-only. The choice trades control and residency for convenience.

The date after which a model's training data ends; it knows nothing newer unless you supply it in the prompt or through tools.